Welcome to the Forum.
You can go above 80 grains of powder if you are shooting a patched .570 diameter roundball.
For example, the Lyman "BLACK POWDER HANDBOOK & LOADING MANUAL" shows the breech pressures they found for shooting a 80 grain GOEX 2Fg powder load under a Lyman #575213PH, 566 grain Minie' as 8,900 psi. That bullet had a Muzzle Velocity of 1058 and 1407 ft/lb of energy.
A patched .570 diameter roundball with an 80 grain load of GOEX 2Fg had a breech pressure of 5,200 psi and a muzzle velocity of 1288.
Upping the roundballs powder load to 120 grains raised the breech pressure to 7,900 psi with a MV of 1568 fps.
That may be somewhat extreme and it may not give good accuracy but it does demenstrate that shooting the 276 grain .570 diameter roundball lowers breech pressures and increases muzzle velocity.
Some folks find the patched roundball is more accurate than the Minie's too. You can play with the patch thickness to try to improve accuracy.
With the Minie' bullets, they must be .002-.003 smaller than the bore size to get good accuracy and often finding a bullet that is the correct size is difficult.
Yes, during the Civil War the Minie's often had more clearance than .002 but precise shooting was not a requirement on the battle field.