I shoot 80 Gr. 2F Pyrodex with a TIGHTly patched .535" RB. I can shoot all-touching groups at 50 yards all day long. Same powder load with the Lee Improved Minie #533-410 opens up the group just a tiny bit. All from a 1:48" Investarms barrel.
You mentioned the RWS #1075 caps. I found years ago that the RWS #1075 is really a #10 3/4 cap - a little bigger than #10, and a little smaller than a #11.
While giving plenty hot ignition, you may be getting just an RCH ignition delay time as the cap is seated fully onto the nipple by the same stroke of the hammer that then sets it off. It may delay ignition just enough to affect your accuracy. Caps are the only thing you haven't mentioned changing.
Try a tin of CCI or Remington #11's, or, as time permits, install a musket nipple & some Musket caps.