Kodiak,
MD is correct about what may be the cause of this in your barrel. If the Muzzle Face is not perpendicular to the bore, it will cause this. This is due to just as the ball is coming out the barrel, the gas pressure would be pushing against part of the barrel that is sticking out further and have nothing to push against on the low side. That would push the ball towards the low side of the muzzle face JUST as it leaves the muzzle of the barrel. However, there are barrels that still do this even when the muzzle face is perpendicular to the bore and that is when the bore is not centered inside the barrel. This last is not unusual as even with modern precision machinery, there are times the bore will not be exactly centered in an octagonal or even round barrel.
(This is definitely one time where modern and muzzle loading rifles are the same, so I hope the moderators will excuse me giving examples from modern rifles.)
HOWEVER, I spent 23 of my 26 year career in the Marine Corps as an "RTE" or NM Armorer and can tell you from long experience that on MANY standard and NM M1 Garand and M14 rifles where the barrel WAS mounted correctly and the muzzle face WAS parallel to the bore, we had to move the front sight ALL the way to the left or right side to get "Mechanical Zero" on the Rear Sight OR at least be within a few clicks of it. OK, I probably should explain that.
With the standard M1, M14 or M16 Service Rifles, if one is only shooting at 100 to 300 yards, it is often the case the front sight can be centered and the rear sight has enough clicks of "windage" or movement to the right or left for any distance between 0 and 300 yards. On some rifles, the rear sight has to be moved a LOT more for windage than on others to get your groups centered on the target. On some rifles, the rear sight may have to be moved 12-20 clicks right or left at 300 yards. However on some rifles, if the range goes to 500 yards like the Army used to do through the 1960's and the Marine Corps still does, you had to move the rear sight so far to the right or left to get groups centered on the target, that there were no adjustment clicks left when the wind came up from one side. In those cases on the M1 and M14 where you can move the front sight right and left, Armorers moved the front sight to make up for how the rifle shot off one side to the other. However, even when Armorers did not have to move the front sight, the rear sight may have been so far to one side and the shooter never realized it when looking at the rear sight.
We set up NM rifles differently for the "Iron Sights," though. The shortest range we shot was 200 yards and the longest was 1,000 yards. Therefore, we needed our rear sights to have the maximum amount of movement to make up for strong winds at 600 and 1,000 yards. So, we centered the REAR sight on mechanical zero and moved the front sight to the right or left so the group would be centered on target at 200 yards. That meant that sometimes we had to move the front sight ALL the way to the left or right and very infrequently we still had to put a few clicks of windage on the rear sight. However, that gave us more clicks to move at longer ranges when the wind came up. So it is NOT unusual to see a NM M1 or M14 with the front sight ALL the way to the right of left side of the front sight base. So in my career, it was VERY common to see extremely accurate rifles with the front sight moved all the way to the right or left.
When I build or rebarrel a NM M1 or NM M1A, I sight the rifle in at 100 yards and center the rear sight and move the front sight to center the groups on target. Sometimes that moves the front sight pretty far to the left or right. On a rifle that will only ever be fired at 100 yards maximum, I normally center the front sight and just move the rear sight to center the groups, though.
On either modern or Muzzle Loading Rifles where you shoot 50 yards or less, most of the time you can center your groups on target by having the front sight in the middle of the barrel and the rear sight moved towards the left or right. That is on most rifles, but not ALL rifles. On rifles that are zero'd at 100 yards, you may not be able to move the rear sight enough to center your groups on target and that's when you HAVE to move the Front Sight to center your groups on target. On a good number of modern and ML barrels, the front sight may actually be a little off from center (if someone already sighted it in) and many folks would not realize it by just looking at it, though it would show up when using a precision machinist's rule or dial calipers.
So, what's the "bottom line" to all this? IF the front and rear sight are not exactly in the center of the barrel, BUT you still get good groups and centered on target, how much does that matter to you? Yeah, it may look a little strange, but does it matter when you align the sights and shoot in the center of the target? If it really bugs you, then you may want to have it checked to see if the muzzle face is not perpendicular to the bore and pay to have someone correct that, if needed. If it is a new gun and still under warranty, you may want to return the gun. Personally, I might do that if it was a new gun under warranty. However, I have learned it does not matter where the front and rear sights are on a barrel as long as the barrel shoots good groups and can be centered on target at whatever range I want to center the groups.
Gus