Yes, it's you. Fortunately, it can be overcome by practice and learning about your rifle. What is the state of the crown at the muzzle? Is it too sharp? Have you tried the over-the-powder wads? Are you wiping between shots? Are you using a wet patch lube or a dry lube? Is the front sight reflecting glare by being in the sun? Try some sight shaders. A tube from a roll of toilet paper can be cut along the cylinder and placed over the sights to eliminate glare from the sun. Is it windy? Concentrate on the front sight. Fill the rear sight with JB Weld (you can apply heat to remove it later) and make a narrow rear sight notch. Verify that the sights are tight in their dovetails.
Heed all the above instructions on practice and follow through. Always rest the rifle at the same place on the sandbags. Try some shots at the entry thimble. Try some shots with the rifle resting on the bags at the muzzle. Make sure the butt rests at the same spot on your shoulder.