Is this off the bench or offhand/freestyle? The load has to be able to do the job off the bench before it can be expected to shoot well offhand.
I have one smoothbore that likes and average to tight load, .600 ball with. 010 patch, the other won't group a tight load worth a darn with a tight load.l, shoots better with a .595 ball and .010 patch than with same ball and thicker patch or .600 ball and thin patch, but,,,, it shoots even better with no patch, just sandwiching the ball between a lubed wad and an overshot card. This is all off the bench. Can I shoot as well off hand, no, but the potential is there, and most people (even with modern guns) can't shoot up to their gun's potential.
When load testing I would put something on the gun to use as a rear sight, or figure out how to best line up the tang screw, just to make sure you are holding the same way every shot.
After the group is good off the bench it becomes a matter of gun fit and mounting the gun to the shoulder and cheek weld the same way every time. Your eye is the rear sight, so, if the gun is on a different place on your shoulder and/or your cheek weld is different on the stock from shot to shot, you are changing the rear sight.
At 25 yards, maybe the slight change doesn't show, but at 50 it does. Or, you are moving the gun to look where the shot went too soon, not following through, due to lack of confidence at 50.