Hazrat Maqsim (may Allah have mercy on him) says that when he causes you harm or disobeys you, then sever ties with him.