Sayyiduna Abdullah bin Abbas (may Allah be pleased with them both) says: There is no sin upon him if he goes around them voluntarily, and if someone leaves it, there is no harm either.
Imam al-Tabarani recorded it in al-Mu'jam al-Awsat. Its chain contains al-Abbas bin al-Fadl al-Ansari, who is matruk (abandoned).
Hadith Referenceمجمع الزوائد ومنبع الفوائد / كتاب الحج / 5531