Amr bin Shu'ayb narrates from his father, and he from his grandfather (may Allah be pleased with him), that the Messenger of Allah (peace and blessings be upon him) said: "There is no zakah on stones."
All the narrators who transmit this hadith from Amr are da'if (weak).
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الزكاة / 7590
Al-Hakam narrates that Ali (may Allah be pleased with him) said: There is no zakah on gems.
This is munqati' (its chain is broken) and mawquf (a saying of the Companion).
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الزكاة / 7591
Sa'id bin Jubayr (may Allah have mercy on him) said: There is no zakah on stones, except when the gems are for trade; nor on rubies, nor on pearls, nor on anything else, except gold and silver.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الزكاة / 7592