It is narrated from Sayyiduna Mu'adh bin Anas al-Juhani (may Allah be pleased with him) that the Messenger of Allah (peace and blessings be upon him) said: "Utter harshness, disbelief and hypocrisy is this: that a person hears the caller of Allah calling to the prayer and inviting to success, and does not respond to his call."
Hadith Referenceالمعجم الكبير للطبراني / باب الميم / 17529