Sayyiduna Abu Hurairah (may Allah be pleased with him) narrated from the Messenger of Allah (peace and blessings be upon him), he said: "I intended to order the young men to gather bundles of wood, then I would command for the prayer and the call to prayer would be given, then I would burn the houses of those who hear the call to prayer but do not come to perform the prayer."
Hadith Referenceمسند اسحاق بن راهويه / كتاب الصلوٰة / 161