Sayyiduna Anas (may Allah be pleased with him) said that the Prophet (peace and blessings be upon him) would not return to his family at night; he would only come back in the morning or the evening.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الحج / 10369