Mujahid (may Allah have mercy on him) said: Some people who were blind, lame and in need would follow certain men to their houses. If those men did not find food for them in their own houses, they would take them to the houses of their parents, which they had built alongside their own houses. Those who were taken along disliked this and said: "They take us to houses other than their own." So Allah Almighty revealed this verse concerning it: "There is no blame upon you" [Al-Baqarah: 236], "food is lawful for them wherever they find it."
The Shaykh (may Allah have mercy on him) said: Feeling hardship despite the owner's consent is not right, because with the owner's consent the hardship has been removed.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الصداق / 14602