Abd al-Razzaq (may Allah have mercy on him) says: I heard a man ask al-Thawri (may Allah have mercy on him): Who are the family of Muhammad? He said: People have differed about this. Some say that the household is meant, and some say: whoever obeyed him (peace and blessings be upon him) and acted upon his Sunnah.
Abu Bakr says: I think Abd al-Razzaq (may Allah have mercy on him) said: 'whoever obeyed him'.
Imam al-Bayhaqi (may Allah have mercy on him) says: Whoever holds this second view would most likely argue that Allah Almighty said to Sayyiduna Nuh (peace be upon him): "Load into it a pair of every kind, and your family" [Surah Hud: 40], and he said: "My Lord, indeed my son is of my family, and Your promise is true, and You are the most just of judges." (Allah) said: "O Nuh, he is not of your family; indeed, his deeds are not righteous" [Surah Hud: 45-46]. So Allah excluded the son of Sayyiduna Nuh (peace be upon him) from the family of Nuh (peace be upon him) because of shirk.
Imam al-Shafi'i (may Allah have mercy on him) answered this by saying that the words of Allah, the Blessed and Exalted, "He is not of your family" [Surah Hud: 46] do not mean what has been stated; rather, they mean that he was not among those fit to board the Ark. Allah Almighty said: "And load your family into the Ark, except those concerning whom the decree of destruction has already gone forth." [Surah Hud: 40]
This verse too makes clear that the son of Sayyiduna Nuh (peace be upon him) was of his family, and the reason he did not board the Ark was that his deeds were not right.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الصلاة / 2869