You guys may already know this but I'll say it anyway.
There are basically two kinds of hard woods.
There are "open grain" woods like Walnut, Oak, Hickory and Mahogany and there are "closed grain" woods like Maple, Birch, Beech and Poplar.
The closed grain woods do not have noticable open pores and will sand/scrape very smooth.
The open grained woods will always have very noticable openings where the pores of the wood meet the surface. Mahoganys grain is notorious for catching on sandpaper and lifting up and tearing a hole in the sanded surface if you sand "against the grain" and catch one of the open ends of these pores.
Fortunatly for us gun builders, I have never known Walnut to do do this.
When sanding Walnut or other open grain woods, you will never be able to remove the pores. They are a part of the way the wood grew.
When the surface is as smooth as possible with 220 grit sandpaper it is time to apply the finish to the wood.
If you like the "open grain" look, just apply the oil or whatever finish you want.
If you want the surface smooth, you must "fill" the pores.
Several different fillers are commercially available to do this, or, if you saved the saw/sanding dust you can mix it with Linseed oil to form a very thick paste.
Rub this paste across the grain to fill the pores and let it dry.(With Linseed, this takes a few days). Then using a rough cloth (like burlap) rub the excess filler off wiping
across the grain, not "with it".
This will leave a filled, smooth surface which is ready for the final finish coats.
Never sand "across" the grain of the wood. It will always leave rough scrape like marks on the surfce. The only way to remove these is to resand the area "with" the grain of the wood.
If the surface has large noticable waves where the shape departs from a smooth surface, you may have "gouged" the wood out making a pocket or hollow. This is one of the dangers of using a hard object like a file to "back up" you sandpaper.
IMO, the only time a person should use a sanding block or other hard tool to back up their sandpaper is when they want to keep areas flat with sharp edges where the surface meets another surface. For other surfaces, I much prefer to let the sandpaper blend and fare into the other surfaces.
If you have created a gouged or hollowed area, the only cure for this is to work the adjacent surfaces down to blend the hollow in.
If you do this, don't just blend the immediate area into the hollow. Work the whole area down gradually to blend with the hollowed area. The further back you can start this blend, the less noticable it will be when the finish is applied.
It's just my opinion, but I've known people to sand their bare wood down using 320, 400 and even the 600 grit paper. This is a colossal waste of time and energy. All wood, including the closed grain woods have pores which are much courser than even 220 grit paper so using the finer papers can never make the surface smoother than the woods pores will allow.
Actually, using these super fine grit papers will only smash the wood fibers down making them look and feel smooth until you wet the wood and "whisker" it down.
Save the fine papers for working the final finish down smooth.