Sayyiduna Jubayr bin Mut'im (may Allah be pleased with him) says that the Prophet (peace and blessings be upon him) said: "O Banu Abd al-Muttalib! (or he said: O Banu Abd Manaf!) If you become custodians of this House, you must not prevent anyone from performing tawaf of it, nor from praying at any time of the night or the day."
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الصلاة / 4412