Obviously, Longbow, much depends on the caliber and gauge of the rifle and smoothbore. There are modern shotguns( smoothbores) that shoot at targets out at 200 yds, with some fine accuracy. Certainly it can be done with BL smoothbores, too. But, most shooters would limit the smoothbore to 100 yds, or just a few yards beyond. If you look at tables for the velocity and energy levels of a round ball at these distances, they lose a lot of velocity and begin to drop dramatically with each yard they travel. Its hard enough to accurately estimate yardage. If you add to that problem figuring out how much to hold over the target to hit it at the longer ranges, using open sights, or, sometimes, only a front sight, You get an ideal of the complexities of shooting a smoothbore at longer ranges.
The rifle imparts a spin to the ball, which contributes to its in flight stability, and allows it also to slip through the air, maintaining a little more velocity, and shedding velocity slower than a round ball shot out of a smoothbore. Still, a 200 yd. shot with a PRB in a rifle is a long shot, with many of the same problems you have with smoothbores. The Round Ball is aerodynamically awful, and sheds velocity so fast it too drops significanly when you push the targets out past 100 yds.
There are recorded instances of Riflemen killing enemy officers at 300 yds, with a PRB. No one denies that. But these men had grown up with their guns, shot them every day, and shot them at all kinds of distances. They knew their guns, much better than most current owners do. Such a long shot would involve a lot of luck for most of us, including me! The main reason is I have never been to a range with a 300 yd. range so I could find out where to hold my sights to hit even a man on a horse at that distance.
At 100 yds, you can expect a rifle to group 5 shots in 3 inches, or as well as you can shoot open sights. Put one of those new-fangles " Scopes" on a gun, and you probalby can reduce that group size in half.
With the smoothbore, the groups at 100 yds, are probably going to be closer to 5 inches, and a scope sight would reduce them to maybe 3 inches. There are exceptional guns and loads out their, being shot by exceptional shooters, who can sometimes do better at that distance with open sights, but more of us would shoot much larger groups than the 5 inches I have indicated for an average. Still, a .62 caliber round ball hitting within 2.5 inches of where you aim at a deer, shot broadside, is going to bring home venison everytime, if you are shooting at the middle of its side, behind the shoulder. A .50 caliber RB will do the same out of a rifle, and shoot much closer to the POA.
Smoothbores, as big game guns, are used where shots are going to be shorter than 100 yds, and often closer tian 50 yds. We are talking about river bottoms, wooded areas, ravines, and anywhere else that farmers cannot get a plow into. In these places, the brush and trees are so thick, its difficult to see a deer at 100 yds, much less get a clean shot( unobstructed by branches or bushes) at a deer.
I know of one deer killed with a smoothbore and PRB at a measured 125 yds, that dropped in its tracks. I also know of a deer shot at 175 yds with a modern shotgun shooting a foster style slug. It also died on the spot. A third deer was killed standing on one ridge line by a hunter on the next ridge line over, a shot that had to exceed 175 yds, again with a shotgun slug. So, the ammo can do it, with a lot of luck getting a hit at those ranges.
Pick the gun for the terraine you expect to hunt, and then practice a lot with it.Those heavy round balls can still kill and hurt somneone at well over 300 yds, so always make sure you have a safe backstop before shooting at anything, including game.