YOu have to shoot at targets at 100 yds. or further for the voids, differences in diameter, and differences in weight to become apparent in your groups on paper.
For casual, off-hand shooting, don't bother sorting. For shooting at targets at 25 and 50 yards, off=hand, the same advice applies. If you are shooting at 50 yds. off a bench rest, or using cross-sticks, some-not all--- shooters notice larger groups when they don't sort. There are too many other factors affecting group size from a rest to be able to say that all shooters are affected the same. I have, however, watched very experienced match shooters shoot much smaller groups at 50 yds. than I can, using my guns, and my loads.
If you are interested in the very best accuracy, hanging around with top shooters is a great way to learn how its done. And, if you aren't sure if its YOU, or your load combination, ask the Very Best Shooter you know to shoot a group with your loads in your gun. It can be a humbling experience, but at least you will know where to concentrate on improving your skills.
Some of the best advice I have received has been at shooting matches, where some top shooter was kind enough to tell me what he thought I was doing wrong, and what I was doing right. :grin: