You are correct in your conclusions and incorrect on some assumptions.
First, you can cleanly kill medium and even larger game like elk with a small to medium caliber with well placed shots. Lots of game are what is call thin-skinned and non-dangerous. You can afford to have them live after a shot until they either bleed out internally or the damage to their lungs prevents them from going further. You my lose a few more if your tracking skills aren't "Boone" quality. Elk, in particular, can cover LOTS of ground before they die from those types of shots. If you want to avoid that potential for loss, you will likely want a through and through shot that breaks some bones. Example below many have seen posted already:
Elk (she wasn't overly big) taken with a 54 cal 375 gr bullet over 100 grains of pyrodex at 85 yards.......That had more retained energy and MV than the old 45-70 405 gr bullet (45 cal, 405 gr, 70 grains BP). The bullet broke a leg bone, passed through heart, lungs and lodged on the far side between ribs under the skin. NOT ONE DROP OF BLOOD WAS FOUND AT THE ENTRY WOUND AND THERE WAS NO EXIT WOUND. I first thought "What the hell? Did I scare her to death?" If she had needed tracking much further than the 20 yards she did go, it would have been some serious work as fresh elk tracks were everywhere.
Picture of her and the bullet
View attachment 187378
View attachment 187379
So "In the olden days" tracking skills were likely better and more game was lost than we imagine. Additionally, they took 200 yard shots routinely, hardly even fretted about clean kills, and usually hunted in groups or at least pairs. Having read hundreds of contemporaneously written books and journals, I can attest to that. What was acceptable then, doesn't apply now.
I submit, for your consideration, that a through and through passage of a bullet after breaking some bones will make for a better blood trial and reduce game loss with much cleaner kills. That we, today, do not hunt in groups and do not routinely take the long shots without care to whether we cleanly kill or not.
So for the smaller whitetail strains, shot in the woods up close, you are absolutely correct. A smaller caliber bullet can and will get the job done. Out west in the open and with bigger game, something more is needed.
There was a reason and it wasn't by accident that as Americans moved west, the calibers got bigger and the loads stouter.
Weapons are tools, you likely don't use a framing hammer to nail a brad on a picture frame and conversely use a brad driver to frame a house. One size does not fit all.
See I just gave you a perfect excuse to get some more guns. I personally have them to hunt squirrels to water buffalo from 10 yards to 1000 yards. Now if I can just find the time, money and location to do that. ROTFL!!!!!!