YOu don't mention reading your patches, and all measurements should be to the thousandths of an inch, not the hundredths. We don't shoot .49 balls. We shoot .490" balls.
You need to invest $15.00 and buy Dutch Schoultz's Black Powder Rifle Accuracy System.
http://www.blackpowderrifleaccuracy.com/
Its the best investment you will make, and it will help you diagnose those problems much better.
I believe you have to clean between shots to have any clue what the gun is doing as far as accuracy goes. If you don't clean, you are presenting a different sized chamber and bore for each shot, and that is no way to hit at any distance in a small group. Yes, you can shoot " faster", but not more accurately. Sorry about that. You can't have your cake and eat it too.
YOu have to read those spent patches. Every one of them. They tell you what went on in the barrel. They tell you if you need to change lubes. They tell you if you need to use more lube.
YOu have to mark your Ramrod( use tape until you settle on a load) and load to the mark every time.
You have to be nice to that round ball. Don't pound on it. Don't bounce the ramrod off that ball when its seated on the powder.
It would be helpful if you have a chronograph. Then you can see how loading the gun differently for each round makes a difference in velocity. You can see the Standard Deviation of Velocity widen, or narrow as you get that good load.
In humid conditions, you have to alter your cleaning habits a bit. I recommend running a cleaning patch down the barrel but stopping an inch before the vent hole. Pull the patch out. If the other side is clean enough, just flip the patch over, and this time go down to the breechplug to pull out the crud down there. In humid, hot, weather, you then need a second ( or third patch) to dry the barrel. You should feel a smooth, clean bore with that dry patch that goes down the barrel. That indicates that the barrel is as clean as it was after firing one shot and being cleaned, and that remains consistent through out your shooting, Shot after Shot.
I do recommend shooters use a metal Range rod, rather than a wooden ramrod that fits in the pipes under the barrel. Also, use a hand over hand method for lowering that RR down the barrel. Its the only way to NOT break a wooden ramrod. I see new guys grabbing the very end of a rod and trying to run it down the barrel in one stroke. YOu can watch as the rod bend, almost 90 degrees sometimes, as they try to force the rod down with all their strength. That is how rods break and get stuck through hands and wrists. And, usually, the only motive is that the shooter doesn't want to get his hands dirty. Why can't you take towels, a bottle of water, and some soap to clean your hands and equipment( including the RR) off periodically? I have a roll of paper towel in my trunk, with a gallon bottle of water, and there is always a small bottle of soap in my range box just for these purposes. A spray bottle of windex, or similar cleaner can be used to " spritz" a cleaning patch, or even used as a lube at the range. It then comes in handy with paper towels, to clean your hands, and the ramrod, and jags of crud that inevitably gets on them. I use the towels and cleaning patches to clean around the touch hole, wipe the frizzen, the flint, and the cock, as well as the barrel where powder residue obviously collects. A Cleaning patch with spit usually takes the residue out of the flash pan between shots, just as a matter of routine.
When weather gets colder, or dryer( usually its dryer in sub-freezing temperatures, unless its snowing) you change your cleaning routine. I use alcohol to clean the barrel, and forget about soap and water. I use one patch to clean the barrel, and one to dry it, or at least check to make sure the barrel is cleaned from using both sides of the first one. That second patch then gets some alcohol on it, and is used to wipe out the flashpan, and the frizzen and flint, etc.
You have to really be observant, and thinking in both Spring and Fall, because the temperatures, and the relative humidity can change so quickly, and require a modification of your cleaning and loading techniques. Most new shooters never even think to pay attention to the weather. It matters.
Do check your stock. The hanger for the barrel key should be elongated, and the key should move in any out with hand pressure, and not require a hammer, and drift, or screw driver to get it out. Sometimes stocks swell in the summer, and that can bind those barrel keys that hold the stock to the barrel. When the barrel heats up, the shots go all over. When the gun is allowed to cool down, the first shot or two comes out hitting close together. It will drive you nuts unless you check this, and fix it.
I also remember a case where the tang screw was tightened down so hard, that it was pushing up on the back of the barrel, and causing shots to go all over the place as the barrel warmed, either in the sun, or from a lot of shots fired. Keep the Gorillas away from those screws and bolts on your rifle. Use witness marks to tell you where the screws should be returned to when you put the bolts and screws back into the stock and action. The bolts and screws should be firm, but don't need to be screwed in so much you are crushing wood. The lockplate should be square to the barrel, and not canted by screwing down the lockbolts too much. Same with trigger guards, and tang bolts, or screws.
Sometimes the inletting of the barrel is just lousy. Epoxy bedding compound will fix that in a hurry, and permanently. If you glass bed the back 4-6 inches of the barrel, and the tang and breechplug, it will also strengthen the stock at its weakest point( because of all the wood removed.) If the bedding is lousy, it is usually evident by the fact that someone else can hold the stock, while you can move the barrel back and forth, or up and down in the stock with your hands. I don't see this often with factory built guns, but I have seen it once. Its easily checked.