Sayyiduna Abdullah bin al-Sa'ib (may Allah be pleased with him) says that the Prophet (peace and blessings be upon him) led them in the Eid prayer, then delivered the sermon. He (peace and blessings be upon him) said: "Whoever wishes to stay, let him stay, and whoever prefers to go may go."
Hadith Referenceالسنن الكبرى للبيهقي / كتاب صلاة العيدين / 6223