Dave,
Try same charge but a .015 patch & a .535 ball. That is my most accurate ball for my .54 cal.
Actually .65 grains holds the best group for it.
I use the load you are shooting & that ball for deer hunting, as it is easier to load.
I shot 37 deer with that load in this rifle. (we are allowed 2 a day for 75 day season)
I have built allot of .54 cal rifles & every one of them has shot well with .015 patch, 65 grains of
FF or FFF, and a .530 or a .535 RB.
Keith