Abdullah bin Umar (may Allah be pleased with them both) said that the Prophet (peace and blessings be upon him) said: "Whoever is invited and does not accept has disobeyed Allah and the Messenger, and whoever enters without an invitation enters as a thief and leaves as a raider."
This was said concerning a man who goes in to another, knowing that he is eating, in order to eat with him. And a hadith has been narrated that negates the special exemption that Abu al-Abbas imagined regarding the meal of the Prophet (peace and blessings be upon him) in the story of Abu al-Darda'.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب النكاح / 13412