Your are shooting an overload. Per the Lyman manual for the .54 GPR the maximum loading of 3F is 100 grains. This is on PDF that I tried to post but it will not do so. That may be part of the fouling issue. The other is the .015 patches instead of .018 patches. Have you inspected any recovered patches? That can be very helpful in determining the proper patch thickness and material required for optimum.
My .54 GPR loves 90 Grains of 3F, .530 RB, .018 ticking and TOTW Mink Oil. Before I tried a lower powder charge and didn't achieve the great accuracy I have now.
Second. I know you are interested in finding a load that you can shoot more repetitively without having to wipe the bore. It that's your goal, then try some other patch lube such as spit or something with a cleaner in it. For hunting, stick with a good lube such as mink oil, olive oil/beeswax mixture or bear grease and leave the spit in your mouth and cleaners under your sink. I do strongly suggest you do your final test shooting and sight-in prior to hunting season with whatever patch lube you plan on hunting with.
Personally, I have no issues with running a moist patch that has Windex on it after a few shots, then follow up with a couple of dry patches. Unless one is competition shooting or perhaps doing a woods-walk shoot, it's no big deal to run a patch down the bore after a few shots. Even then I would still choose to run a moist patch down the bore after a few shots if time permits. I can promise you one thing. After I do the above, regardless of how many shots I've fired, at the end of the day and come time to clean, all of my ML's have little fouling that comes out.
Thirdly, Swiss powder has proven to me to be cleaner than at least GOEX. Never tried Gaffs. Either way, Swiss is a cleaner burning powder.
Good luck.
