Flinter, could vary a lot. While you don't need more than 80 grs for deer, I don't know if that's your most accurate load. I have one .54 with a 44" barrel and 1-70 twist (old Green Mountain)that did not reach it's best accuracy until 105 grs of 2F. 110 grs opened the group up again and my usual 90 gr load that works in my other .54's didn't do it at all in this slow-twist barrel.
So, as many have said here, you can certainly go down in powder and still have plenty, or you could go up if needed for the accuracy and flat shooting you want.
As a side note, I'd much rather NOT shoot 105 grs. But I do find 90 grs to give what I call a 100 yard .54 cal gun...a tad high at 50, right on at 75, and just a tad low at 100...all within a range that I can just hold on the chest center and if the shot is done well, it's a dead deer.