Nafi' (may Allah have mercy on him) says that Sayyiduna Abdullah bin Umar (may Allah be pleased with him) used to say: If the call to prayer is made while a man is with his wife, that does not prevent him from fasting, if he intends to fast. He should get up, take a ritual bath and complete his fast.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب الصيام / 8024