It is narrated from Sayyiduna Abd al-Muttalib bin Rabi'ah (may Allah be pleased with him) that he said: Banu Abd al-Muttalib gathered and said: "Why do you not ask the Messenger of Allah (peace and blessings be upon him) to give you the responsibility that he has given to such-and-such a tribe, that is, to appoint you to collect the zakah?" So they met Sayyiduna Ali (may Allah be pleased with him) and spoke to him about it, and he said: "Allah, the Mighty and Majestic, and His Messenger have refused this for you, that you should be given the dirt of people's hands" - or he said: "the washings of people's hands." Abd al-Muttalib (bin Rabi'ah) said to his father: "Send me." And Sayyiduna al-Abbas (may Allah be pleased with him) sent his son al-Fadl. We came to the Prophet (peace and blessings be upon him) and went in, and he met each one of us privately. Then the Messenger of Allah (peace and blessings be upon him) said: "What are you two keeping hidden in your hearts?" I said: "O Messenger of Allah, your people and your cousins have sent us so that you may appoint them to collect the zakah just as you have appointed other people." He (peace and blessings be upon him) said: "Indeed, Allah and His Messenger have refused this for you, that you should eat the dirt" - or he said: "the washings - of people's hands."
Hadith Referenceالمعجم الكبير للطبراني / باب الميم / 17811