Sayyidah Aishah (may Allah be pleased with her) says that the Prophet (peace and blessings be upon him) gave his Companions permission, so they visited the House at midday, and he (peace and blessings be upon him) visited it with his wives at night.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الحج / 9638