Sayyiduna Jubair bin Mut'im (may Allah be pleased with him) narrates that the Messenger of Allah (peace and blessings be upon him) said: "O sons of Abd Manaf, do not ever prevent anyone from performing Tawaf of this Sacred House (the Ka'bah) or from offering prayer in it at any hour of the day or night, whenever he wishes." The wording of the hadith text is from the narration of Ali bin Khushram.
وَالدَّلِيلُ عَلَى صِحَّةِ مَذْهَبِ الْمُطَّلِبِيِّ أَنَّ النَّبِيَّ صَلَّى اللَّهُ عَلَيْهِ وَسَلَّمَ إِنَّمَا أَرَادَ بِزَجْرِهِ عَنِ الصَّلَاةِ بَعْدَ الصُّبْحِ حَتَّى تَطْلُعَ الشَّمْسُ، وَبَعْدَ الْعَصْرِ حَتَّى تَغْرُبَ الشَّمْسُ، بَعْضَ الصَّلَاةِ لَا جَمِيعَهَا
And the explanation of the evidence for the correctness of the Matlabi school of thought: that the Prophet Muhammad (peace be upon him) forbade praying after the Fajr prayer until sunrise and after the Asr prayer until sunset, so by this he meant some prayers, not all prayers.