Hazrat Hudhaifah (may Allah be pleased with him) says that prayer is not broken by anything; however, stop the passerby as much as you are able.