You,
@Golden Child, mentioned that your grandpa built this rifle. Based on the observation that he never shot this rifle; I am guessing that the hooked breech to barrel fit may not be perfect. Since you are using Pyrodex, that almost unnoticed but very corrosive fouling needs to be fully cleaned out. Shooting good groups one day, taking the rifle home, removing the barrel from the stock to clean and replacing the barrel into the hooked breech may not be consistent. Until the barrel gets located in the same place from shooting session to shooting session, there won't be accuracy on the target. As the hook in the barrel is installed in the breech tang, make sure the barrel is fully seated in the breech tang. You may need to use a soft dead blow hammer to tap the muzzle before the wedge is installed. Is there any indication of rubbing at the breech or wood that may interfere with the installation of the barrel?
You haven't fully described your cleaning regimen. What are you cleaning with and what kind of lubricant are you using to protect the barrel after cleaning? Are you cleaning any protective oils from the barrel before shooting? You should be cleaning the protective oils out with rubbing alcohol before shooting. Store the rifle muzzle down after cleaning to allow the protective oils to drain from the T/C chambered breech and flash channel.
The flattening of the nipple at the top is called mushrooming and can prevent a cap firing. It is a simple thing to use a file to remove just the expanded top of the nipple. Since you are using Pyrodex, a nipple designed to send more heat to the Pyrodex is a benefit. Hotshot, Spitfire or Red Hot are brands of nipples (1/4-28 thread) to replace the original nipple. It is possible that there may be some timing issues that you are not noticing, but is enough to allow the movement of the barrel on firing.
It also sounds as if you are using a fixed volume spout on a powder flask. Do get a good volumetric adjustable powder measure. Pyrodex is volume for volume interchangeable with that volume of black powder. Pour the measure of powder from the flask into the measure and only load from the measure.
You do want a moist patch for the ball. The moist patch will soften the fouling in the barrel. I know there isn't much apparent fouling when using Pyrodex, but the moist patch will make the loading easier and there shouldn't be so much liquid that the powder is fouled. When hunting, an oiled patch is better and a dry patch over the powder and then the lubricated patch with ball can be loaded.
Patches for cleaning are different that the patches used for wrapping a round ball. You want an absorbent patch for cleaning. I use patches made from baby diaper material or baby flannel. These are nice and soft and clean fouling from the grooves. You want a patch for the round ball to be tough to withstand the forces imparted during loading and firing. It should hold some lubricant to ease loading and soften fouling. For patching the round ball, material such as pillow ticking of 100% cotton is among the best options. Thickness of pillow ticking patches with the round ball of 0.490" should be about 0.015".