You are correct about that. The Blacktail deer of Victoria Island are quite small, and from what I know no larger than black tail deer found further south, but for Whitetails, his rule holds well. I have shot 350 lb plus whitetails in Alberta, never seen one close to that size in Texas.
Even on Alberta Whitetails, 50 grains will do the job, IF you hit them in the right spot. As I have said before, I prefer heavier loads as they are far flatter shooting and I really can't tell the difference in kick with a PRB and 90 grains vs 50 grains of FFFG. I would be willing to bet that if you had someone else load your gun for you and not tell you which load was which, you couldn't tell much difference either.