You probably already know this, but just in case....
Be sure to bevel the edge of the inlays so that they are wider on the surface than on the bottom. When making your intial cut around the inlay, undercut it a little bit under the bevel. I leave the inlays above the surface and file down to the wood. If you bevel the edge of the inlay, you'll still have a tight fit when you file the surface down.
Remember, practice makes perfect. :v