Imam al-Shafi'i (may Allah have mercy on him) said: "If (the imam) keeps them (at the frontier) for a long time and does not let them return home, he has done wrong, and it is permissible for all of them to oppose him and return (to their homes)."
Hadith Referenceالسنن الكبرى للبيهقي / كتاب السير / Q17848