You obviously just accepted whatever statements were told you about the caliber of the gun, without every actually measuring the bore!
Please, from now on, every time you have a new gun, MEASURE the bore diameter with Calipers, and if you have a rifle, measure both the bore and groove diameters. Also, always check that diameter of the balls you buy, or cast, to see EXACTLY what size they are. The .600 Cal. ball I bought from Track of the Wolf did measure .600, but I did have some variation in weight. The cast balls my brother provides also measure .600, but he sorts them by weight for me.
The same attention to detail has to given to patches. YOU HAVE TO MEASURE the patch thickness, and not rely on what it says on a package. I once bought two strips of patching from the same vender marked the same size, and -- YEP-- They were different. One was as represented, while the other was a few thousandth's different. Its not the vender's fault, but it is the fault of the manufacturer. Someone in my club was looking for some patching, and we found the stuff I had and could not use was just what he wanted. I got my money back that way.
If you buy fabric, you need to wash it and dry it a couple of times to remove all the sizing in it(mostly atarches) and shrink the fabric before taking a final measurement. Some fabric is pre-washed and pre-shrunk, but test a sample before you buy a life-time supply!
If you are going to shoot a bare ball, you will have steaks of lead to deal with in the barrel, and as it builds, accuracy will deterriorate. At the very least, you should use a good overpowder card wad to seal the gas behind the ball so you don't have gas cutting in addition to rubbing on the barrel to deal with. If you lube the barrel after seating the ball on the wad, and covering it with an Over Shot card, or two to hold it in place, try lubing the barrel with a lubed cleaning patch( not a prelubed ball patch) That will help reduce the amount of lead that is rubbed off on the barrel, leave your ball mostly round when it leaves the muzzle, and should give you better accuracy. Another way is to use a pre-lubed wool wad between the overpowder wad, and the ball, but that will soften fouling, but will not do anything about reducing the amount of leading in the barrel. I know its another loading step, but this is a single shot flintlock, not a .45 auto. Accept it. Enjoy the fact that you can't rapid fire the gun, that you have to get close, and that hitting any game with that huge ball is going to put meat on the table.