I am glad the information from the patches confirms what I suspected. I think you are a very good shot, and I was disappointed with some of the targets. Cold or not, you just are not that bad a shot!
That and the dimension you gave to the ball you are loading, without telling us how thick the patch( denim comes in various thicknesses ) led me to think you could do with thicker patches, even if you have to double up with one thin and one thick patch to get there. I think the barrel is very capable of good accuracy, even looking at those groups.
I found that using an OP wad( 1/8 fiber wad, available from Track of the Wolf, or directly from John Walters) protected the Patch( so you don't get what happened to these, here!) AND, allowed the thinner patch to center the smaller ball, to give consistent performance. Its the OP wad that reduces the wide swings in velocity that you get when you are not using a thick enough patch for the ball you have chosen.
There is nothing wrong with using a thin patch. It allows you to load the ball easier- thumb pressure to start it, and then the ramrod to run it down, hand over hand, onto the OP wad. Since the barrel is not rifled, the patch does not have to Grab the ball and impart the spin of the rifling to it.
In rifles, the choice of ball diameter becomes critical, based on bore diameter, and even groove depth. That is not a problem in smoothbores. What is a continuing problem with both rifles and smoothbores is the tearing and burning of patches that are too thin, from gas cutting. Since I use OS cards, or OP wads in my smoothie when shooting shot loads, its no extra expense to use them when shooting RB loads.
When I first shot my Fowler with undersized patches and wads, the velocity was more than 250 feet LESS than when I obtained and used the proper wads and patching. That put my RBs on target about 8 inches low at 25 yards! YIKES!!!
With the OP wad, and correct patches, I not only got better velocity, but the group moved up just under the aiming point at 25 yards, allowing me to move back to 50 yards for further testing. The chronograph showed that I was getting much more consistent velocity when I used the OP wad behind my PRB loads, than when I fired only the PRB.
The patches showed some tearing, and burning when fired without the OP wad, so I knew that I had to go to a thicker patch, or find a larger diameter ball. I have a handful of .610" balls to try in my gun when the weather gets a bit better. I had been shooting .600" diameter balls. I now have a selection of patch material to try, also. And, if I can start getting consistent groups at 50 yards, I will see if my powder charge can be increased ( or decreased to improve the groups.)Only when I settle on the load the gun likes, will I finally begin filing my sights to zero the gun for that load.