My thoughts about this man's problems:
First with 100 grains of any powder, you need a fire wall between the powder and your PRB. I suspect that he is not getting the sealing he needs if he is just using a Patch around his ball in his rifle. He needs to use an OP wad, or a good filler, to seal the gases and get the most use out of that powder charge. And, he needs to be picking up everyone of those patches he fires without the OP wad or filler, to check them for holes, and burning.
I know that with my .50 caliber rifle, which I shot for years with 96 grains of FFg behind a lubed PRB, when I added an OP wad, the POI at 50 yards went up an full inch! My patches have always looked good, but they looked even better after I fired those loads with an OP wads behind them.
Second, the poster doesn't tell us what sight picture he is using. If he is putting the top of the front sight level with the top of his rear sight ears-those right next to the notch -- then he should have had enough play to lower the front sight and raise his POI.
If he is holding a " Fine Bead", where he puts just a hint of the top of the sight in the bottom of his notch, then he's going to be shooting low until he files most of that front sight OFF!( He says he can't file the rear sight notch any lower, indicating that its fairly close to the top flat of the barrel.)
I have my own doubts that the barrel is bent, or that there is anything wrong with its muzzle crown. The consistency of his reported hit locations suggests otherwise.
It would be helpful to know what the MV on these loads are, and the SDV of the loads are for a string of his shots. The SDV would be a good clue to the need for the OP wad or filler. And a jump in the MV when the OP wad is used is to be expected as well. I saw a 25% increase in MV in my 20 ga. fowler when I used the OP wads in my barrel. And the POI rose about 6 inches at 25 yards.