The soldiers were trained to aim for the belt buckle on the opposing horseman, and the gun was designed to shoot to a higher point of impact. That way, if, as pistol shooters are want to do when in a gunfight, they don't get that front site down in the rear sight notch, the ball will hit anywhere from the stomach to the head. A man can duck and be missed by a pistol aimed at his heart or head, but his center of gravity, particularly sitting on a horse, is at his stomach. These guns were not designed to be shot a bullseyes; but rather at human silhouettes. Keep that in mind when you shoot them.
Don't hesitate to reduce the powder charge to bring the poi back down, or to use fillers to keep the ball at the mouth of the chambers. Depending on the size of your hand, the grip of the gun, and the amount of overtravel of the trigger after the sear is released, will often decide whether you have windage problems. Have you fired the gun with your left hand, and with two hand holds? Does it still shoot right? Does it shoot right for other shooters?
Only if the answers to those questions are " Yes ", and there is little to no overtravel in the trigger would I resort to filing that notch in the hammer. I have had one gun that had a notch so obviously off center, and shot so far to the left, that even at 50 feet, it was hitting off the target. That one saw the file fairly early on.