Sayyiduna Abu Hurayrah (may Allah be pleased with him) says that the Messenger of Allah (peace and blessings be upon him) said: "If you say to your companion (during the sermon): Be quiet, even then you have engaged in idle talk."
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الجمعة / 5824