Imam al-Shafi'i (may Allah have mercy on him) said: "If (the imam) keeps them (at the frontier) for a long time and does not let them return home, he has done wrong, and it is permissible for all of them to oppose him and return (to their homes)."
Sayyiduna Abu Firas (may Allah have mercy on him) said: Sayyiduna Umar (may Allah be pleased with him) addressed us in a sermon and said: O people! I do not send governors to you so that they beat your bodies and take your wealth; I send them so that they teach you your religion and your Sunnah. If any of them does anything other than this, bring his case to me and I will take retribution from him. Beware! Do not beat the Muslims and so humiliate them; do not drive the Muslims to disbelief by repeatedly scolding and denying them; and do not keep the fighters from returning to their homes, for if you keep them, they will fall into trial; and do not ruin the fighters by putting them on difficult paths.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب السير / 17848