Shot lots of deer with both, and it always "seems" the deer go down a little quicker, especially as the range stretches past 50 yards, using the 54. Very subjective, because I don't feel like they die any too slow with the 50 either.
Haven't shot elk with them myself, but guests on our place have killed several with each. Long and short of it, the guys with the 50's are all shooting 54's now.
Though I'd use something even bigger for moose, I'd surely pick a 54 over a 50 if I only had the two. Lots of moose killed up here every year with both, but the guys packing 50's are almost all stuffing conicals rather than RBs, while most guys using 54's are using RBs.
One measurable difference I notice, and my wife notices even more, in identical models the 54 is always a little lighter and a little less muzzle-heavy than the 50. For that reason my wife is a 54 shooter.