Unless you're trying to shoot the fleas off a dog's arse at 500, I wouldn't worry about it.
If it does concern you overly, don't remove the barrel or bed it to minimize any displacement when replacing. Which might very well open another can of worms.
Shoot it as is for a while, carrying on any...