Changing patches may solve the problem but first you really need to shoot a few shots with your current load.
Then you need to find the shot patches and examine them.
The outer edges will be tattered but the area where the ball was seated against the bore should be uncut and have no burned material.
The area that was behind the ball may be dark brown or it may look almost like it did before it was shot.
If there are rips or tares where the patch was against the rifling, that is the problem.
Tares or rips can happen when the ball/patch is being loaded at the muzzle, while it is being rammed down the bore or it can happen when the gun is fired.
You really should examine the rifling at the muzzle.
Most new barrels have a bunch of sharp edges there.
If your rifle barrel has sharp edges in that location, get yourself a piece of the black, wet/dry sandpaper in 320 or 400 grit.
Tear a piece of it off and press it down into the muzzle with your thumb. Then rotate your thumb and the paper back and forth to sand away all of the sharp edges. This should fix part of the problem.
If your using very old pre-lubed patches, the lube may have caused the cotton fibers to weaken so try some new patches.
Make sure they have enough lubtication on them and try shooting again. Again, examine the patches for rips and tares and burned areas.
Your gun really should shoot better than you say it is and I think if we are given enough information about your findings we'll get it working like it should.