That rear sight is standing about 1-1/4" above center bore, and the rear sight is only about 1/3" above the center bore... do you not realize how extreme of an angle that creates with the bore (provided the bore is remotely centered, which it clearly isn't)? Going to a lower rear sight and no front sight (the way he's headed if he files it anymore) isn't going to solve that. OP also stated that the rifle is shooting great with the load he's using, so stabilization isn't an issue. There is something mechanically wrong with the barrel itself.
It might need shimmed and/or bedded at the muzzle, or the barrel may need bent slightly to put the end of the bore where it belongs; and it doesn't usually take much, a matter of a degree or two).
Anyway, get a new front sight, so you have some adjustment once you're done shimming and/or bending it. Rear sight? that's up to you, just make sure the front works with the rear as far as height is concerned.