Imam al-Shafi'i (may Allah have mercy on him) said: "There is no harm in it, just as he hires from them (the disbelievers) their camels, their houses and their slaves; and whatever he pays to them, or to the ruler as their agent, is not a humiliation for him; rather, it is a debt upon him which he pays."
Imam al-Shafi'i (may Allah have mercy on him) said: "As for the hadith narrated from the Prophet (peace and blessings be upon him): 'It is not proper for a Muslim to pay kharaj, nor for a polytheist to enter al-Masjid al-Haram,' what is meant by it is only the kharaj of jizyah."
Imam al-Shafi'i (may Allah have mercy on him) said: "Some people of piety and religion have taken kharaj land, while some people disliked it out of caution."
The Shaykh (Imam al-Bayhaqi) (may Allah have mercy on him) said: "As for the dislike, it is based on the narration that follows."
Hadith Referenceالسنن الكبرى للبيهقي / كتاب السير / Q18394