I also suspect that the bore butter is some of the issue, but patch thickness could also be some of it if you're not running a very tight fit? I store with barricade and don't bother trying to swab it out before loading the first shot, for the protection it offers as I will leave mine loaded the whole season if needed. My first shot is always lower than my next shots, even though I swab every shot, so the fouling after the first shot is tightening up the fit and increasing the velocity I think. My windage is always good on the foul shot, so it basically works out that my foul shot is a 50 yard zero and my shots after that are a 75 yard zero (my preferred zero). If your foul shot is fine windage wise, perhaps this method would work for you too? My PRB fit is pretty tight too, a very firm smack is required on the starter.
The suggestions of using a bit thicker patch for the first shot are probably pretty good ideas if your foul shot is just low but fine windage wise.