You may have differences when casting from lot to lot if you use different batches of lead but using the same batch of lead, if your casting technique is good, you shouldn't get more than a grain or two from ball/bullet to bullet. Even 3 grains isn't going to hurt you.
I've already weighed jacketed bullets from reputable manufacturers and when you get up over 150 or so grains you will notice a weight variance of a couple grains.
So being realistic, how much does your accuracy improve when weighing and sorting your balls? 1/2" at 50 yards, 1"? How much at 100 yards? I would consider 99.9% of our shots during an actual hunting scenario is <100 yards. IMO weighing cast balls from the same lot is an effort in futility.
But if it makes you feel good and makes your confidence higher then by all means go for it. Its your time and like reloading, it could be a method of self relaxation.