It is narrated from Abu Huraira (may Allah be pleased with him) that the Prophet (peace be upon him) said: "When the Imam is delivering the sermon and you say to your companion sitting beside you, 'Be quiet,' then you yourself have engaged in idle (useless) talk."
Hadith Referenceمسند الإمام الشافعي / كتاب الجمعة / 427