It is narrated from Sayyiduna Jubayr bin Mut'im (may Allah be pleased with him) that the Prophet (peace and blessings be upon him) said: "O Banu Abd Manaf! O Banu Abd al-Muttalib! I will surely come to know that you did not prevent anyone from praying at this House at whatever hour of the night or day he wishes."
Hadith Referenceالمعجم الكبير للطبراني / باب الجيم / 1586