It is narrated from Hazrat Aisha (may Allah be pleased with her) that the Prophet (peace and blessings be upon him) used to pray in those clothes in which he would have "privacy" with his family.
Hadith Referenceمسند احمد / مسند النساء / 25822
Hadith Gradingحکم دارالسلام:صحيح لغيره، وهذا إسناد ضعيف لانقطاعه لأن سليمان بن موسى لم يدرك عائشة