Ata bin Yasar (may Allah have mercy on him) narrates from the Prophet (peace and blessings be upon him) that he said to Quraysh: "You are the most entitled of people to this matter (leadership) as long as you are with the truth, unless you turn away from it; then you will be stripped just as this palm stalk is stripped." He (peace and blessings be upon him) pointed to a palm stalk in his hand.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب قتال أهل البغي / 16546