Unless your bore is undersized, you should first try using thicker patching materials. Try .012, .015, and .018" patches. Lube them with different lubes, both liquid( Hoppes #9 Patch lube and cleaner, olive oil, vegetable oil, a " dry lube " of 1:6 Ballistol to water, letting the patching dry in the sun to evaporate the water, but leave a thin coat of oil in the fibers, water, spit, etc.) and then try greases and lubes,( like Wonderlube, or Stumpy's Moose snot recipe, or a 50/50 mix of melted beeswax and olive oil, etc.)
I think 80 grains and that thin patch you are using is a bit much. Are you locating and examining your spent patches on the ground after each shot? Are you "reading them?" At 50 yards, your gun should be putting RB in one ragged hole all day long. Back that load down to caliber- 50 grains of FFg, and start there.
In my GM .50 barrel, I found that a .490 ball, a .015" patch, lubed with Young Country 101, now Wonderlube, or Bore Butter)and 60 grains of powder gave me that one hole group at 50 yards. My 100 yd load is 75 grains of powder with the same Ball/patch combination. My barrel is 39 inches long.
Make sure you are using a SMALL aiming point on that target each time you fire- a "sighter" as its called in chunk gun matches.
I use a rectangular piece of card stock, turned 45 degrees, so that I have a .45 degree angled "arrow" pointing towards where I want the balls to impact. I can center that point on my front sight, and bring the sight up until the point is covered and then fire, for very small groups.
I know one man who uses the same diamond approach, EXCEPT, he has the diamond above his POI, and brings his sight up to the point, holds it and fires. I haven't tried his technique, but it should be as good or better than what I have been doing for years.
If you are not an experienced shooter-- no insult intended here, 'cause I just don't know you--- use a solid bench, good snadbag rests, front and back, and use your thumb and index finger to Pinch the trigger, so that only your shoulder takes the recoil when the gun fires.
Don't wrap your hand around the wrist of the stock when testing your loads, because this is the most common source of erratic groups even when the powder charge, and Patch/ball /lube is correct.
To raise the gun up or lower it down, move the rear bag, or simply pinch the rear bag to push up against the toe of the stock.
Let the forestock rest on the forward rest without your hand there. You are testing the equipment, not your ability to hold a gun off-hand. That comes later.
