It is narrated from Hazrat Jabir bin Samurah (may Allah be pleased with him) that I heard a man asking the Prophet (peace and blessings be upon him): "Can I pray in the clothes in which I go to my wife?" The Prophet (peace and blessings be upon him) said: "Yes, unless you see a stain on them, then wash it."
Hadith Referenceمسند احمد / أول مسند البصريين / 20825
Hadith Gradingحکم دارالسلام:هذا الحديث موقوف على الصحيح، وفي سنده عبدالله بن ميمون، فهو مجهول لكنه توبع