Read those spent patches. They will tell you a lot about your barrel, and accuracy problems. Heavy charges of black powder( 70 grains and more), and the substitutes will almost always shoot better with a Over Powder( OP) wads used as a " Firewall " and gas sealer.
Those heavy Round Balls will fly through most any deer you shoot out to 100 yds., and you don't need to try to send your shoulder back into the next country to help that ball do that job. If you didn't know it, "Recoil" negatively affects accuracy. When you are hunting game with a single shot rifle, and have to reload it in the field to have a second shot, the " Name of the game" is accurate placement of that FIRST SHOT. :hmm: :thumbsup:
Hitting that target with the first shot, every time, is more important than making tight groups
at 100 yds. But, the better the gun and he SHOOTER'S ability to make small groups, the better the indicators are that recoil is not spoiling the game. Do some comparison penetration tests with That first load, and with the "lighter" load you find is more accurate at 100 yds. You will find that the weight of a round ball is responsible for most of the penetration, NOT THE VELOCITY.
If you study tables on velocity, you will find that velocity of any RB drops off quickly from the muzzle, because the Round Ball has such a poor Ballistics Coefficent( drag factor). You can lose 40% of your Muzzle Velocity by the time the ball reaches 100 yds. The other sad fact is that the faster the ball leaves the muzzle, the faster it slows down. Increasing velocity is a "compromise" between trying to get a flatter arc of your trajectory, and getting more "power" at the longer range. No amount of powder is going to make your gun, shooting Round Balls, shoot flat to 200 yds, for instance. If it were possible, neither the gun, nor your shoulder, could take the recoil generated by such a load for long.
There are plenty of suppository rifles available that can shoot BULLETS fast enough to be reasonable flat shooting for longer range targets. Some of them even load Black Powder. There are some "slug guns"( heavy MLer bench guns) That are designed to shoot lead BULLETS out to long ranges, too. These guns are too heavy to be shot from the shoulder, and therefore are NOT suitable for most hunting. :hmm: :hatsoff: :hatsoff: