Imam Malik (may Allah have mercy on him) also narrates the hadith in the same way, with the words "when the caller fell silent". Ibn Shihab (may Allah have mercy on him) added that when the imam came out they would end their prayer, and when the imam spoke they would fall silent.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الجمعة / 5685