Snakebit has given you the most accurate information on the Crockett. Much of it will depend on how you hold the rifle. In fact, it means everything with the Crockett. Its not difficult to learn but until then, you will have consistency issues. The issue will be multiplied whenever shooting while standing and, or trying to shoot on unlevel ground when getting a rest from a tree. It will also rare its ugly head whenever shooting from a bench. However, after you figure it out, its gravy. Not difficult, just a little different.
The other highly probable reason would be your particular patch material. Patch materials are not created equally. Patch material may also break down after it becomes old. This became apparent and was a hard learned learned lesson a few months ago. I was using some ticking of, I think, was .018" thick. It appeared to be good. However, after some accuracy deterioration, I recovered some fired patches. Sure enough, they had holes burned through them and/or were blown apart. Ordered some new .018" ticking from October Country and the issue was resolved. It became obvious that the old ticking I had was inferior, either in thread count or due to age. Do not skimp on patch material. Buy the best you can get and avoid WalMart material.
Secondly, if you were, in fact, shooting .021" thick ticking and couldn't find any of of those fired patches at only 20 grains, that also suggest poor or inferior patch material. I can and have shot 30 grains of powder and it did not blow up the patches. Not only that, but when I tried .021" patches in one of my Crockett's, I practically had to have a hammer to seat the load. With a .310 RB, ~20 grains of 3F and .018" ticking, you should be able to obtain good accuracy out of your Crockett, as long as you have sufficient patch material. Snakebit and I, and I think Gum Slew Muzzleloader, all shoot the same exact loads out of our Crockett's.
.310 RB
.018" ticking.
20 grains of 3F Swiss.
TOTW Mink Oil.
Where I differ is with his statement that it will clog up the breech area if you push a damp cleaning patch too fast down the bore. I push the damp patches down the bore the same as I do any other time and I have not experienced any issues whatsoever. Sometimes, during a hunt and after I have already shot a squirrel, or shot at one, I run those damp patches down the bore and reload as fast as I can. I use moist/damp cleaning patches sprayed down with Windex, followed by one or two dry patches. Now mind you I have put no less than 500-600 shots through my one Crockett. It's had different size RB, different size thickness and/or different material of patches, along with different powder with different powder charges, and different lubes. I do find it conducive to run a damp patch down the bore after every other to every 3 shots, on average. Sometimes I go as many as 5 shots. One day when I was still using Bore Butter, I shot 11 times and did not run a damp patch down the bore. However, I was using .010" patches.
One last addition, in terms of shooting patches. As long as the patches are cut large enough for proper stabilization of the RB, I doubt you will see any difference if you were to cut them square, or at the muzzle. However, considering the small size of the .32 RB, not cutting them round might be more conducive. I use two loading blocks and it makes loading much, much, easier. One thing to avoid is to not cut the shooting patches too large for the .32 Crockett. Every time I have, I had problems with the patch/ball wanting to pull back out after I seated them, as if they stuck to the jag.
Also, I suspect you might find a difference in accuracy, as well as less fouling, if you can get yourself some 3F Swiss. I know I found a difference overall. Schutzen is said by many to be dirty burning powder. Swiss is about as clean burning as you will find. Swiss is also proven to be a little hotter burning powder.
Patch lube can make a difference from one ML to the next even of the same make and model. I was lucky to get mine to shoot fairly accurately with Bore Butter, which despite what some may claim, I have never once had an issue with the stuff with any ML. However, since it is no longer being produced, I moved on to TOTW mink oil. At this time I'm out of the third can of it but before I open another, the plans are to try some of the real bear grease and real bees wax mixture that I made up. No problems with TOTW Mink Oil whatsoever, but I do want to see how the bear grease works out.
Good luck.