Semisane: The weight of that .54 ball has more to do with how much penetration you get on a soft-skinned animal like a deer, than velocity. I am sure that the ball will penetrate and kill a deer at ranges beyond 100 yards, with that light load. However, ball drop becomes a problem as the range increases, and many shooters just don't know their guns well enough to know how much to " hold over " their sights in order to place that ball where it must strike to kill the deer. I have seen enough deer with broken legs brought into deer check stations to prove that.
So, the question probably should be, HOW FAR are you capable of hitting a deer with that light load? I met a man deer hunting one year who showed me where he was standing when he shot a deer the last day of the season, late in the afternoon, the prior year, and pointed across a ravine to where the deer was standing. The distance was at least 150 yards! He dropped the deer with a shot to its spine! Pure luck, but he was still " Proud " of the shot he had taken. He was shooting a smoothbore shotgun, no rear sight, and shooting foster style slugs. He used the same gun the day before he described this long shot, to kill a deer at about 10 feet! in very dense new growth.
The typical Whitetail is going to show about 10-12 inches of chest on a broadside shot. A Center hit will be 6 inches up or down, then. That light load is probably going to drop at least 4 inches at 100 yards, and maybe more. You can add or subtract inches depending on whether you sight the gun to hit POA centered, or are using a 6 o'clock hold to get a center hit. Throw in range estimation errors, poor light conditions due to weather, and time of day, or fog, and missing the deer at 100 yards becomes a real possibility. That is before we consider human factors unique to the shooter, such as flinches, "buck fever", poor mounting of the gun due to excitement, etc.
The reason more powerful loads are recommended is because the shooters who recommend them have been through the experience, and understand the value of the flattest shooting trajectory they can get for any caliber rifle they shoot. Overcoming all these unplanned for conditions by having a gun that can shoot further, and still hit a small target, in spite of them, is what gets the job done, ethically.