Crow, you want to make the bottom of the inlay smaller than the top sorta like putting a tapered cork in a test tube. But do keep the taper small, I’d eyeball it at about 5-ish degrees.
It should take a while, in other words, don’t rush it, take off only wood that shows indication of your...