Imam al-Shafi'i (may Allah have mercy on him) says: This, like the two hadiths before it, concerns the polytheists in general terms, but what is meant by it, and Allah knows best, is the idol-worshipping polytheists. In the presence of the Prophet (peace and blessings be upon him), or near him, there was none of the polytheists of the People of the Book except the Jews of Madinah, who were allies of the Ansar; and the Ansar had not all entered Islam when the Prophet (peace and blessings be upon him) first arrived. So the Jews made a truce with the Messenger of Allah (peace and blessings be upon him), and no hostility towards him appeared from them in word or deed until the Battle of Badr; on that occasion some of them spoke out in enmity towards him and incited others against him, so the Messenger of Allah (peace and blessings be upon him) carried out killing among them. To my knowledge, there were in the Hijaz only Jews, or a few Christians of Najran; the Magians were in Hajar and in the lands of the Berbers and in Persia, far from the Hijaz, and apart from them there were many idol-worshipping polytheists.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب السير / 18627