Sayyiduna Hudhayfah bin al-Yaman (may Allah be pleased with them both) says: I did not take part in Badr for this reason: my father Husayl and I went out, and the disbelievers of Quraysh seized us. They said: You intend (to go to) Muhammad (peace and blessings be upon him)? We denied it, (saying) that we wanted to go to Madinah. So they took from us the covenant and pledge of Allah that we would go to Madinah but would not fight alongside him (peace and blessings be upon him). We came and told the Prophet (peace and blessings be upon him), and he (peace and blessings be upon him) said: "Go to Madinah; we will keep the promise, and we will seek Allah's help against them."
The Shaykh says: This is because their turning back did not entail abandoning an obligation, since going out was not obligatory on them, nor did it entail committing anything forbidden; whereas returning to the disbelievers and living among them is not permissible when one fears trial for oneself in returning, and Allah knows best.
Hadith Referenceالسنن الكبرى للبيهقي / كتاب السير / 18429