Sa'id bin Jubayr (may Allah have mercy on him) said: There is no zakah on stones, except when the gems are for trade; nor on rubies, nor on pearls, nor on anything else, except gold and silver.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الزكاة / 7592