All things considered, seems like the best accuracy is going to be obtained with the patched ball, even if some are getting "good" results with bare-balls. Certainly a more convienient loading procedure.
If you are using denim now, there's no need to get another ball size, you can just try thinner patch material. Denim is thick stuff, I can shoot .570" balls out of my .61" calibre Jaeger with it.
I get 3" groups from a Brown Bess carbine, with rifle sights. I use a tight ball/patch combo, but not as tight as yours. So you should be able to use a thinner patch, I think that a tight combo is good, but there is a point where tighter stops making any difference, accuracy-wise. One whack on the short starter gets the ball into my barrel, and then it rams down reasonably well. Get about six shots maximum before wiping, but normally wipe after every third shot. I figure that's perfect for hunting. That's with a .735" ball, .750" bore, and a .010" patch and a Wonder wad, and I load the first load, into a clean barrel with a .018" patch.
The 3" group was obtained wiping (actually cleaning) the barrel between each shot, and using the .018" patch. This will stay on the paper plate out to 70 yards, giving me a 70 yard kill-zone, under perfect conditions. (steady rest, animal not alerted or moving, and no wind)
Rat