the shape of the barrel (i.e. round, octagon, some combination of the two, or any one of a number of variations) doesn't really control what happens on the inside of the barrel.
rifled barrels have grooves in them which impart spin to the projectile when the gun is fired. this spin is what stabilized the bullet (or roundball) as it goes down range. smoothbore barrels do not have the lands and grooves- they're smooth all the way down the bore.
here's where the terminology gets tricky: sometimes the gun is referred to as a 'rifle' when it actually has a smooth bore. most smoothbores have only a front sight, like a modern shotgun. a 'smooth rifle ' also has a rear sight, so it will look like a rifle, but it actually has no rifling in the barrel. the two most common calibres are .54 (smooth) and 20 guage (which is very nearly .62 calibre). other smoothbore barrels can be had, so you could make a 'smooth rifle' in just about any calibre you want.
even though they do not have rifling, smooth rifles are fairly accurate. the accuracy of a smooth rifle will be about the same as that of a rifled barrel (all else being equal) out to about seventy five yards. the advantage of having a smoooth barrel is that you need not use roundball- you can put birdshot or buckshot in there and go after a wider variety of game.
hope this helps. (by the way- if you're looking at a 20 guage smooth rifle, i'd say go for it, but that's free advice and worth every penny).