Mujahid (may Allah have mercy on him) says: When this verse was revealed: "And whoever seeks a religion other than Islam" [Surah Al Imran: 85], the followers of all religions claimed to be Muslims. Then Allah Almighty revealed this verse: "Pilgrimage to the House is a duty upon people for the sake of Allah" [Surah Al Imran: 97]; so the Muslims performed Hajj and the polytheists did not.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الصيام / 8609