The wood won't. The dust and grit on the wood will. My carved-from-one-piece maple stub-starters are doweled only 1-1/2" long and they seldom touch the barrel walls. I haven't used my oak, witch hazel or aluminum 5" or 6" starters in years.
With reasonable care you shouldn't need worry over it. The danger is when you give it a side-slap and run it across the muzzle on the way down. Solution: use a shorter short starter, a looser fit patch, or slipperier lube. You can also intentionally cone the muzzle, which is like being worn but very, very evenly around the entire muzzle, and you can then just push the ball in with thumb pressure. I compromised and had a "medium radius" crown that funnels the patch in easier. I can use a 0.010" undersize ball with 0.018" patching and no starter at all, just choke up on the rammer to get it started.
I've known benchrest shooters who won't use anything but one-piece steel cleaning rods (that includes brass, alumnium and fiberglass) because they say it is the only choice that grit won't imbed itself in. But they are somewhat fanatical and trying for five shots in a bore-size hole.
I just wipe my rammer off frequently and keep it waxed up (Minwax Hard Finishing Wax)and clean.