A bore is a bore and a ball is a ball, the accuracy going out that bore is entirely dependent on the accuracy of the stuff going in it. The whole "this caliber/chambering is more accurate than that caliber/chambering, simply by virtue of it being X-chambering/caliber" is about as true in Muzzle loading as it is in modern guns.
The larger bore is going to be more expensive to feed (both in powder and shot), and paper can't tell the difference between a 45gr .31 ball and a 224gr ball. The recoil will be more for the larger guns (requiring more powder and shot), and that generally isn't ideal for target shooting. If you are shooting round balls at paper, most likely the distances aren't going to be far enough for the BC advantage the larger ball has (roughly .074 for the .54 vs .043 for a little .32) to warrant the extra cost, weight, and recoil, IMO. Cost is a major consideration when it comes to competition shooting, as much of your performance is directly related to how much quality trigger time you can put in.
While many people will insist that you "need" a 1:66 twist for round ball to achieve anything resembling accuracy, I would like to submit to the court the fact that many barrel makers throughout history used faster... sometimes much faster; and still got very good accuracy (3/4 turn in the length of the barrel was common on the 30ish inch barreled Germanic rifles). A faster rate of twist requires less velocity to achieve a stable RPM for the ball (lessening recoil and cost in powder). Once again, paper doesn't care how fast the ball's going. A "good" twist for a .32 would be a 1:28, .36 would be 30, .40 would be 40, .45 would be 48, 50 would be 50, and .54 would be 55. Slower rates of twist require higher velocity to achieve a stable RPM, and that increases costs and recoil.
The one real exception to the above points would be in knock-over steel competitions, where the velocity and mass of your ball can make the difference between scoring a point and getting zip. Then again, I have always felt that those type of competitions are more angled towards hunting-type guns, rather than being simply the pursuit of accuracy.
As far as patching is concerned, you can either use a bore sized "belted ball" (essentially, a groove sized ball sized down to bore diameter) and no patch at all (which was done); or you could use cloth (a tightly woven cloth is desired, as it resists abrasion better than a loose weave, pillow ticking is often used, but silk and linen have also been used) cut into a square/triangle/circle that has enough material to envelope the back and sides of the ball. Another option is Paper patching your ball (not only is this cheaper, but it allows you to use a ball that is much closer to bore diameter, more on that in a second). To use paper patching for a ball, you essentially make a very thin paper sabot on a bore sized dowel, with a cavity on the end to accept a ball, a glue stick, and a tiny bit of thread to keep the ball from pushing through the paper cup. Another way to paper patch (or cloth patch) would be to use an "X" shaped patch (or two strips crossed, that are slightly under bore diameter in width). Shooting-quality Paper is roughly 10x cheaper than cloth, substantially more easy to find, and it's not like you are going to reuse it.
I have found that the easiest way to achieve accuracy with a ML rifle is to use the thinnest patching possible, and a bullet that is as close to bore diameter as possible. This minimizes the amount of deformation the projectile must go through to upset into the rifling. The less the projectile has to deform to take up the rifling, the less of an impact slight variations in the hardness and density of the projectiles has on pressures and velocity. The purpose of patching in a rifle is NOT to "seal the bore", as anything you can easily blow air through isn't going to hold 20,000 PSI
; rather, it is to provide a more durable bearing surface than the very thin line around the diameter of the ball that is going to be in contact with the barrel steel. If the bearing surface is inadequate to resist the incredible amount of shearing force (from the rotation of the rifling) and friction in the bore, the projectile will "skip the rifling" and not spin adequately/consistently. To this end, I have also found that "shallow groove" rifling (roughly .003"-.005" deep grooves), combined with paper, cup-style patching and a .01" under-bore-sized projectile to be easiest to achieve accuracy in. My only "dedicated" round ball rifle that I still own is a former Tradition's PA Pellet (1:48" twist) that has .005" grooves and is loaded with 100gr weighed charges of 3F powder and paper patched balls. With the crude sights Tradition's put on that gun, I get average 1.73" groups at 100 yards from a rest, and that's with that hunting load, I still need to work up a lighter target load with it (which potentially can be more accurate, for various reasons).
For accuracy purposes, I also recommend pre-weighing out your charges on a scale, and bringing them to the range in vials or paper tubes. Black powder of a particular formulation gives the same amount of gas for its weight, every time (obviously allowing for some lot to lot variations). BP, being irregularly shaped, is prone to not being volumetrically metered out as consistently as more symmetrical granular propellants (think: flake smokeless vs ball powder). Also, I recommend looking into using a drop tube, in order to ensure that the charge lands in the breech area the same way, every time; as opposed to some of the powder getting caught on the rifling down the barrel, and settling in the breech area a different way each load. As far as loading is concerned, I have found that compressing the load in the barrel leads to inconsistency, as there is no way you are going to put the same amount of pressure on that charge each time, by hand. Even if you use a tool like a precision packer (most trad competitions don't really allow those, in my experience), you are still not going to compress it the exact same way each time. I have found that just gently sliding the ball down the barrel until it just touches the powder is the most consistent way to load. I would compare charge compression in a ML to a combination of cartridge OAL and neck tension in a modern cartridge, and those are very important for consistency.
Another thing to consider is a set of pin gauges to monitor the erosion of the vent, as once a vent gets to a certain point of erosion, it tends to erode at a much faster rate. A rapidly eroding vent leads to wild pressure and velocity swings, which robs accuracy. During the development of your loads and practice, periodically check the diameter of the vent, and take note of where the accuracy started to fall apart. Change out your next cone before it gets to that point. They're cheap, and are a consumable component of shooting.
Last thing is sights, for a purely target rifle, I would recommend a Tube sight (think a scope without glass, and an aperture at each end) and use "rabbit ear" mounts for a Malcolm Scope with the fine adjustment screw kit. Second best would be a hooded, dual aperture sighting set up; with a tang-mounted Veneer sight rear (preferably with an adjustable iris installed (they had them, look at Pedersoli's Hadley style eyepieces)) and something like a Lyman model 17A front (get a couple of different sized aperture inserts for different lighting). Third would be a plain blade front sight and a simple rear-mounted aperture sight. Fourth would be a fine bead-on-a-post front sight, and a semi-buckhorn rear sight with a 3/4 circle notch (sized to make the bead appear to fill the notch when sighting). Fifth would be a tight rear notch and plain blade front. Dead last would be a large "U" notch rear and a "Barleycorn" front.
Obviously, a set trigger is a plus. For an off-hand gun, proper balance and stock fit to you and your style of shooting are critical. I, for one, Use a "swiss hold, and that means that I not only can get away with a more butt-heavy, light weight rifle, but also require less mass out at the muzzle for a rifle to not feel overly heavy. That stance allows you to brace your supporting arm against your ribcage, rather than trying to hold the rifle up purely with your arm muscles. A rifle that is primarily going to be shot standing can afford to have more drop at the comb than one that is going to be shot in various positions (a hunting/general purpose rifle). A hooked butt plate is a nice thing to have as well (not the SMR or "Hawken" style meant to be shot off the bicep, the Continental European style meant to be shot from the shoulder), as it frees up your dominant hand to purely manage the trigger, rather than holding up the back end of the rifle. Look at Germanic "Scheutzenfest" rifles (off-hand shooting competitions typically shot at 40 rods (220 yards)) for an example of dedicated off-hand rifles.