Abu Qaza'ah (may Allah have mercy on him) said that Abd al-Malik bin Marwan, while performing tawaf of the House of Allah, said: May Allah Almighty destroy Ibn al-Zubayr (may Allah be pleased with them both)! He lies against the Mother of the Believers (may Allah be pleased with her), saying: I heard her say that the Messenger of Allah (peace and blessings be upon him) said: "Were it not that your people's time of disbelief is so recent, I would have pulled down the House of Allah and included the wall in it, for your people cut the building short." Then al-Harith bin Abdullah bin Abi Rabi'ah (may Allah have mercy on him) said: O Commander of the Believers! Do not say this, for I heard the Mother of the Believers (may Allah be pleased with her) narrating this. So he said: Had I heard this before demolishing it, I would have left it upon the building of Ibn al-Zubayr (may Allah be pleased with them both).
Imam al-Shafi'i (may Allah have mercy on him) said: I heard a number of the people of knowledge from Quraysh mention that about six cubits of the Ka'bah, up to the wall, were left out.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الحج / 9316