Urwah says: I saw that once Sayyiduna Umar (may Allah be pleased with him) came to the Black Stone and said to it: I know that you are a stone, you can neither benefit nor harm anyone. If I had not seen the Prophet (peace and blessings be upon him) kissing you, I would never have kissed you. Saying this, he kissed it.
Hadith Referenceمسند احمد / مسند الخلفاء الراشدين / 381