Sa'd bin Ubaydah (may Allah have mercy on him) says: I was with Abdullah bin Umar (may Allah be pleased with him). I got up and left a man of Kindah with him, and I went to Sa'id bin al-Musayyib (may Allah have mercy on him). He says: The Kindi man came in a state of alarm and said: A man came to Ibn Umar (may Allah be pleased with him) and said: "I have to swear by the Ka'bah; shall I swear (by it)?" He said: "No, rather swear by the Lord of the Ka'bah, for Sayyiduna Umar (may Allah be pleased with him) swore by his father, and he (peace and blessings be upon him) said: 'Do not swear by your father; whoever swears by other than Allah has committed shirk.'"
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الأيمان / 19830