I totally agree with you.
I have found that using new 220 grit paper, applying only a very light pressure to it works great for removing whiskers without removing any wood from the surface.
One of the tricks here is to always sand (or scrape) against the direction that feels "rough".
Which direction is this?
To find out, after the water has dried, lightly rub your hand in both directions along the length of the stock.
One direction will show little resistance. The other direction you will feel thousands of tiny fibers trying to keep your hand or finger from moving along freely.
Once this resistant direction is determined, do all of the sanding (scraping) in that direction.
This will cut off or shear off the thousands of little fibers leaving the wood smooth and clean.
If you sand or scrape in the direction that shows little resistance, the sandpaper or scraper will just push the little fibers back down against the surface, leaving them behind so they can again "rise to the occasion" if they get dampened again.
It depends on the wood but on many stocks, this dewhiskering needs to be done several times before applying a stain or aqua-fortis.
When no new fibers rise after dampening the wood and letting it dry, it's ready for the final coloring and oiling.