Well at 80 grains in a .54 you're right about in the middle range of powder. Not many would consider it light, nor would people consider it a heavy load. So would you gain a whole lot more with 20 or 40 more grains of powder?
You would gain some, but if its not enough to make a difference to you, don't worry about it.
I use 120 grains. I'm something like 2 inches high at 50 and right on at 100 yards. With your load you might be comfortable hitting a deer at 100 yards. But what if your range estimation is short by 20 yards? And what you thought was a 100 yard shot is really 120 yards.
This is a decision you have to make for yourself. If your gun can stay inside the kill zone out to 100 yards but drops severely after that, you have a few ethical options. Make a commitment to pass on all shots beyond 75 yards (then if you make and error and that 75 yard shot turns out to be 100 the deer will still die quickly). Or carry a rangefinder, or pre-sight in your various shooting lanes.
So more velocity gets you flatter trajectory. It also gets you a harder downrange "hit". And it gets you less wind drift. And it gets you less bullet flight time. These are all advantages, and those advantages show up stronger in uncontrolled situations.
For example. At a shooting range, 2" trajectory rise won't matter much. In the woods I have missed deer because I aimed under a branch, forgot to consider the rise, and hit the branch.
Bullet flight time won't matter shooting at a still target. At a running deer however, slower bullet means you have to lead farther. Not that I'd recommend shooting at running deer with a muzzleloader. But I have had instances where I pulled the trigger and at the slight moment before my vision was blinded by flash & smoke, I swear I saw the deer leap forward. The ball did strike farther behind than I expected it to.
For many hunters this "gain" is not enough to be worth the "cost" of extra recoil, expensive powder, potential loosening of groups.
Now, if you're shooting at 80 grains, like I said you're about in that middle. So you may not appreciate adjusting powder charges for plinking.
However, if you're preferred hunting load is 120 grains like mine, it certainly makes sense to drop to a 60 grain charge when the target is a mere charcoal briquette at 25 yards.
My old club in Eau Claire, WI used to do a silhouette match. There were iron silhouettes at 25, 50, 75, 100 & 125. The 125-yard targets were big iron bears. That was where these small advantages played out. The guys shooting .45 calibers would just make the bear ring but not fall over. If you were using a .50 or .54, you could get them to fall . . .if you used enough powder. One guy brought a .72 and that thing flattened the bear like a Peterbuilt truck!
P.S. Keep in mind this all implies that your gun can manage heavier loads without a great loss in accuracy. My 120 grain charge still groups under 1.5-inches at 100 yards. Not all guns will do that. Some the accuracy goes to pot quickly.
