It is narrated from Sayyiduna Mu'adh bin Anas al-Juhani (may Allah be pleased with him) that the Prophet (peace and blessings be upon him) said: "It is enough wretchedness and failure for a believer that he hears the mu'adhdhin repeatedly calling to the prayer and does not respond to his call."
Hadith Referenceالمعجم الكبير للطبراني / باب الميم / 17531