There is a major statistical difference between small caliber round balls and large caliber round balls when traveling down range. You can verify it with a book like the Lyman black powder handbook, 1st ed. a 36 caliber ball loses approximately 64% of it's muzzle velocity by the time it reaches 100 yds. A 40 caliber ball loses approximately 60% A 45 caliber ball loses about 55%, a 50 caliber round ball loses about 50% a 58 caliber round ball loses about 42% of it's muzzle velocity at 100 yds. There is some variation depending on charges, etc, balls moving faster than sound tend to lose more velocity by 100 yds the faster they start out.. This inverse proportion is not a hard rule, but is close to the trend if you examine the velocities in the book.
Now I don't believe for a second that ft pounds of energy down range is the end all, but since energy is a factor of velocity squared, the smaller the ball, the more energy loss by 100 yds. A 58 caliber ball not only packs more energy, it makes a hole 3 or 4 times the size of a 40 caliber hole. ie more blood loss and shock. Better chance of an immediate kill and certainly a better chance of a blood trail.
Some people like to compare it to fishing and using 2 pound test line. The difference is if the fish breaks the line, it gets away barely injured. Better than a mammal dying days later of sepsis from being gut shot.
I owe it to the game to kill as quickly as possible. That can mean only close well placed shots or somewhat less well placed at greater distance with something still likely to kill.
I don't know how good a marksman a person is. No doubt some could take an elk with a 32 caliber. But by far, most could not. If I were to be fortunate enough for a large game hunt, I would want every advantage possible. A moderate hunting load in a 54 or 58 caliber does that for me without sacrificing accuracy..