Sayyiduna Abu Hurairah (may Allah be pleased with him) said: The Messenger of Allah (peace and blessings be upon him) said: "When the Imam is delivering the Friday sermon and you say to the person sitting next to you, 'Be quiet,' then you yourself have engaged in idle talk."
Hadith Referenceسنن دارمي / من كتاب الصللاة / 1587