Actions which do not belong to the Prayer, invalidate it
Question :
Is it correct that excessive movements during the prayer invalidate it?
Answer:
There is no doubt that excessive movements which are not part of the prayer, such as walking, frequent glancing around and lots of vain, needless movements of the hands invalidate it, and contradict its wisdom, and purposeless movements and actions in prayer are proscribed, even if they be a little, because it is evidence of negligence and is inconsistent with the required humility.
Source:
Ibn Jibreen
Fatawa Islamiyah, Vol. 2 Pages 189-190