I've got this problem solved in the flintlock. After a ton of experimenting I got it. The original load was with a .018 cut at muzzle patch and a .530 ball. When I switched the patch to a pre-cut oiled .020 all the shots were in the same group. The first shot from a clean barrel and any subsequent shots on the dirty bore are in the same group. This was repeatable.
I did try .535 balls with the original .018. It groups ok, but not as good as the .530 with the same patch and the point of impact is about 2-inches lower. The .535 with the oiled .020 patch shoots nearly the same point of impact as the original load or the .530 with oiled .020, but the group is not as good.
Bottom line, I can either load the first clean bore barrel with the .530 and .020 oiled patch and then the rest with .018, or just use the .020 for all the shots.
Interesting that the tighter, and tightest loads are not the most accurate.