Sayyiduna Ibn Abbas (may Allah be pleased with them both) said: The Hijr (of Isma'il) is part of the House of Allah. Allah Almighty has said: "And let them circumambulate the Ancient House" [Surah al-Hajj: 29], and the Messenger of Allah (peace and blessings be upon him) performed tawaf from behind it.
Hadith Referenceالمعجم الكبير للطبراني / باب العين / 10942