@Fishslap, were the groups on the targets you posted all shot on the same day? I find I only shoot well (especially with heavier charges) for so long. As the range session goes on I start to get sloppy and lose focus. The second target looks like that might be the issue.
You had three shots touching just to the right of the circle with one shot in it. The low shots aren't off by much and even the high shots would be in the vitals assuming you're going for a heart/lung shot. The main reason I would prefer the higher charges is for the increased impact velocity on longer shots. A .54 punches a big hole but I prefer to get some expansion to create a larger wound channel.
With my percussion GPR I average 1450fps with a 70gr charge of FFF Goex. 85 grains of Goex averaged about 1550fps and 95 grains came in about 100fps faster near 1650fps. All velocity readings were with the chronograph 11 yards from the muzzle. A ball sheds velocity fast and while there might not be much difference between the impact velocity at 100 yards between the light and heavier load, I would still want the higher velocity load. This gun is also more accurate with FFF than with FF.
Another benefit is that the heavier charge will shoot flatter. This can help if your estimated distance is off a bit. Using a ballistics calculator there doesn't seem to be much difference in drop between a 70 and 95 grain load at 100 yards. I definitely see a big difference between the two when I'm shooting. It seems much more than the ballistic charts say it should be.