I think this is a real concern. Many people, when they hear talk of .54 or .62 caliber pistols, think something that big can routinely stop anything on the planet.
I have two large bore, single shot pistols. One is a .62 caliber flintlock smoothbore, 12" barrel and the other a .577 percussion with a nine inch barrel. Both are quality firearms and both are accurate. Neither is relevant for hunting, except as a back up gun. When friends see them for the first time, they often comment that they would stop an elephant. When I tell that they are actually far too anemic to be used on even a white tail deer, I usually get a disbelieving reaction--anything with a bore that big just has to be serious medicine. I tell them that the .62, when loaded with shot, does make a fine rabbit gun out to maybe 15-20 yards. If the rabbit is sitting still.