Id hate to spend the money if it isn't going to do what I want it to do
So how do you feel about hunting squirrel with a modern .410 ? Your .50 in a smooth barrel is a tad larger so should be a bit better on small game, though modern .410 shotguns tend to be choked.
a .410 shotgun shell normally has ½ ounce of shot. So that gives you the following pellets in the following sizes when the shot is lead...
#4 68
#5 85
#6 113
So if you kept that weight of shot, you'd need to pattern your barrel to see what size shot flies with the best pattern, and thus give you the best chance at a harvest of the animal.
However, with the thickness of a .50 barrel, and the fact that in the past some hunters would launch two patched round balls in a "bear load"..., that barrel can probably take without any harm a heavier shot load. Two .490 round balls would equal about .80 ounces. So I'd say, you can go up to a ¾ ounce, without worries. So your pellet amounts would be close to a "light 20 gauge load", and in numbers would be...
#4 102
#5 128
#6 170
That's not too bad and launched with 50-60 grains of 2Fg, with a wad of tow and the shot in a paper "cartridge", should be good to 20-25 yards or so.
LD