Try using a Walter's Fiber Wad under the patch and ball. Also, try lubing the barrel with a patch after seating your PRB. Sometimes these two things will help you gain consistency when nothing else seems to help. Most of us fiddle with patch sizes and Ball diameters forever to find the right combination for a gun. If you use a chronograph when working on loads, you will find more quickly what is working and what isn't, by the keeping track of the Standard Deviation in Velocity from one load combination to another. I have found that using fiber wads, or a filler gives me more consistent burning of the powder, a cleaner barrel, and more consistent velocity, with the resulting lower SDV that causes. Group sizes go down quickly. I was shooting 50 grains of 3Fg powder with a .451 ball in my .45 rifle, but it was made by a different company. That may affect your choice of ball diameter and patch thickness( I used .010" patching) but the powder charge should be good out to 50 yds. Recently, I was helping my brother with his gun, and we found a big difference in powder when we sifted his powder to remove all the fines. The 3Fg that remained gave much more consistent groups, although the velocity dropped a little after removing all the 4Fg "fines" from the powder, as would be expected!