Because when the person praying turns his attention here and there during his prayer, Allah Almighty also turns His blessed face away from the face of the one praying.