First I want to say that I have nothing against shooting spikes or any other deer of the hunter's choice and as is legal. But, I don't think it's necessary to target yearling spike bucks to have a healthy natural herd.
Deer herds are full of inbreeding. While yearling bucks may disperse from their mother's home range, Dad may still be there breeding generations of his daughters, etc. We would have a LOT more spikes if the main cause was as narrow as inbreeding.
A spike buck may be the result of inferior genetics but it could also be from other factors such as a late birth that affected it going into it's first winter reducing it's first antler growth as a yearling, or poor minerals in an area affecting forage value. A hard and lean first winter could also have an impact. And, a spike could be superior genetics as some buck fawns have grown hard spikes in their summer of birth, though these are probably only an inch or two long.
I remember a fenced in buck at a Christmas Tree farm we used to go to. The first year they had Rudy he was the scrawniest looking little spike I've ever seen! Year two he looked healthy and had a good rack for his age. As he got older his rack really turned on, as usually happens going into their 5th year since bone growth is completed and more minerals now flow to the rack. He went non-typical and at his peak had a massive non-typical rack with 19 pts and a bulky mature buck body.
I believe most bucks, regardless of their first rack, will grow a substantial rack given age and nutrition. Do genetics help? Of course! Those folks on deer farms determined to genetically manipulate and produce "artificial" deer have proven that. But given time and adequate nutrition, most unaltered natural bucks will develop a very nice set of headgear.