Sayyiduna Anas bin Malik (may Allah be pleased with him) says that the Messenger of Allah (peace and blessings be upon him) said: "In the time of Adam (peace be upon him), the site of the House was a handspan or more, as a mark and sign. The angels used to perform Hajj to it before Adam (peace be upon him). Then Adam (peace be upon him) performed Hajj to it, and the angels came to meet Adam (peace be upon him) and asked: O Adam! Where have you come from? He said: I have performed Hajj to the House. They said: The angels performed Hajj to it before you."
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الحج / 9835