Hi Dean,
I have a CVA .54 St. Louis factory made Hawken that I've had for 3 years. It's a very good shooter!! At 50 yards, .530 round ball, .015 pillow ticking, 50 grains of FFF, I have hit 14 of 16 shilloutes at 50 yards!! That's a 'group' of 3 to 5 inches or less.
#1 Consider that you have a brand new barrel!!! It took me 100 to 150 rounds before my loads went down with consistancy. It was part training myself what the barrel and gun were doing and part what I should be doing. It takes a while for the barrel to get seasoned - just like cast iron cookware.
#2 From everything I've learned every barrel has a different 'personality'. You have to learn what your barrel 'wants'. After shooting 1,500+ rounds, I can do a walk through, shoot 15 rounds , not have to clean, and hit 12 or 14 targets out of 15 if my 'head' is together.
#3 Are you doing the same exact thing each time? Powder load, patch, lube, ball?
#4 And the big one!!! What is your sight picture and shooting procedures? The original sights on the rifle were hard to get the same sight picture every time. Being 1/32 of an inch off on your sight picture means a lot at 50 yards. I went to Track of the Wolf and order better, more original styled buckhorn sights which helped a lot!
And the hardest thing to admit - How's your vision? I had to admit that I was having problems seeing the sights even though I have 20-20 vision at 50 yards but wear reading glasses.
If you haven't already done it, join a black powder shooting club. Talk to the good shooters. They can give you some great tips.
AND the most important fact is the " O. E. FACTOR" YES, that's the "Operator Error Factor" - ME!!! If I don't have it together, I can't hit the side of the barn!!!
Hope this gives you some ideas.
Keep your powder dry!