A big problem many have is called a "Peek". It's wanting to look at the target (or game) to see if it's been hit,, with a right hand shooter a peek will move the muzzle left as we lift the head a bit and naturally move what's in the way of our vision, out of the way. (the muzzle)
That's just peek, there is also a push or a pull (different) and drop not to mention flinch.
Sure the argument is always that everything happens to the projectile and muzzle in a fraction of a second and it does,,
But it's the anticipatory move that cause the rifle to move before/as we shoot.
It's a real thing, and can be observed by someone watching the shooter.
If someone never shoots with other people, or doesn't record what they do from a spectator view it'll never be noticed.
Follow through is something that has to be practiced.