Square up the back first or before you go even further. You have a straight barrel, so no damage has been done at this point. You probably do have more barrel showing at the rear because they lift the router up when it gets back there resulting in that area of the bottom flat(s) usually being a little higher. Once you get it squared and the barrel mating at the rear, then work all of your flats as you go. You can work on the sides a little as you have done in order to get the barrel into the stock - but don't overdo it. Based on your picture, it actually looks like you have a fairly good contact on the bottom flat for a good portion of the channel - you are not going to get much better no matter how much you try, but you do want it even (color/contact transfer) for the entire length. Once that is done - make sure your highest exposed area of the side of your side barrel flat is at least half way down into the inlet and that the rear end of the barrel is below the wood line at the back where you will inlet the tang. Then, if you have a portion of the wood on the sides that is too high up the barrel flat - you can measure and lay out a line with a long ruler and file that wood away making sure to keep a straight wood line down the entire length. I do recommend you draw file the barrel before you start inletting also. Hope this makes sense and can help you. Ask more questions if you would like and someone will help.