Sayyiduna Abdullah bin Mas'ud (may Allah be pleased with him) said: The Messenger of Allah (peace and blessings be upon him) said: "When you are three people, then two of you should not whisper to each other leaving the third alone, because this will cause him distress."
Hadith Referenceسنن دارمي / من كتاب الاستئذان / 2692
It is prohibited for two people to whisper privately with each other in the presence of a third person, because this may cause the third person to suspect that perhaps they are speaking ill of him, or that they are planning or conspiring against him, or that they do not consider him trustworthy, and this will hurt his heart.
However, if there are more than three people, then there is no harm in whispering privately, but a person should still avoid situations that give rise to suspicion and doubt.
Allah Ta’ala says: «﴿إِنَّمَاالنَّجْوَىمِنَالشَّيْطَانِ﴾ [المجادلة : 10]» “Whispering is a deed of Shaytan (Satan).”