You can read numerous articles/studies/notes/thesis that state to kill a deer you require a minimum of 800 foot pounds of kinetic energy on "impact" to ensure enough "shock", "tissue damage", and "blood letting" to effect a kill.
Obviously, PRB shooters (who couldn't make 800 ft pounds at 100 yards with a 58 cal PRB pushed by 140 grains of power) and a traditional bow hunter who is lucky to have 40 pounds of KE at impact are relying "solely" on "blood letting" to effect the kill (with an arrow shaft "smaller" than a 32 cal ball, plus the blades of course, but still at about 10% the KE of a 32 PRB at the same range).
A hole through the lungs with a 32, 36 or 40 will let blood just the same as a hole from a 45, 50 or 54.
If you are close enough, the small calibers will go through and through just as well as the larger calibers will at greater distance.
If you miss the vitals at 100 yards with a 54 the deer will wander off and die injured possibly many days later.
If you put a 32 PRB through the heart at 25 yards the deer will fall down in short order.
I don't like BBQ'd goat AND will stick with my 40
I do understand that small cal's are not for everyone or even the majority.
But just because "some" believe they are inadequate does not mean they are - it simply means they are inadequate for the way they hunt or their own abilities/practices.
If the caliber is legal and you are willing to do the work/practice to get proficient with it, then I say use it.