I always have trouble with patch box inlets swelling so the PB's never fit as flush as they did before finishing. Yes, I usually have to scrape out the edges to get them back in there again, (but this can be mitigated during the finishing process somewhat by scraping finish out of there before it gets a chance to get hard). This seems to apply to the finials more so than the lids, which fit somewhat looser anyway.
When I inlet PB's to start with (on the bare wood during the build), I get them really close to perfectly flush with the chiseling, but the final "flushing" I so with the file and sand paper, which involves taking just a little bit of metal away. Once that's done I take them off for engraving and leave it off for finishing. After the wood is finished and done, it's reinstalled, but the edges (of the metal) are always somewhat proud to the wood.
With permanent inlays, (like estrucheons, cheek stars, muzzle caps, thumb pieces etc.) I leave them on the gun for both engraving and finishing. Because of seasonal wood movement, it seems there is never a time they are ever perfectly flush again. Either the wood is proud, or the metal is. Just the nature of the beast.