It is narrated from Sayyiduna Ibn Abbas (may Allah be pleased with them both) that the Prophet (peace and blessings be upon him) said: "O Banu Abd Manaf, if you are entrusted with this matter (the custodianship of the House of Allah) after me, do not prevent anyone from performing tawaf of this House or praying at whatever hour he wishes, by night or by day."
Hadith Referenceالمعجم الكبير للطبراني / باب العين / 11311