SMO: Try using a bore brush to physically get all the crud out of the barrel, and just don't rely on soap and water, or other solvents. I do like Hoppes Black Powder Solvent, and used it with my first rifle to good effect. I have not tried the newest version of that solvent, yet.
You may have some corrosion in the barrel where that crud built up, if it was allowed to sit for any time before you cleaned it out. It happens. Use a wet, double patch on a cleaning jag to make a tight fit in the barrel and then " feel " for any roughness at that place in the barrel. If its there, that is part of the problem you are having with the crud building up again.
Remove the roughness by wrapping steel wool over a brush, and working it back and forth in the barrel. I also like to use a lapping compound on a tight patch, or some of the members here have had good luck using those green scrbber pads sold by 3M in the grocery store for cleaning pots and pans. One way or another, you want to lap the barrel to reduce or remove the rusted area where the crud build up. The lapping will not destroy accuracy, and may in fact help it. As long as the last 8 inches of barrel behind the muzzle are tight, you are going to get good accuracy.
I read someone else's post that talked about Elephant powder. If that is what you have been using, try Goex, or Swiss, or some of the other real black powders. AND, last of all, you may as well consign yourself to cleaning that barrel between shots. You got into this problem trying to shoot without cleaning, and the only way to avoid it in the future is to clean. Some people, like Roundball, live in areas where its both hot and humid enough to keep fouling soft. He only has to use a true solvent during his winter months. His winter is 3 months. Mine is 5+ months. Where I was raise, winter was good for more than 6 months of the year. If you have lots of wind, its likely that your relative humidity is not going to be very high, and your residue will dry quickly in the barrel.
Don't rely on directions found in manufacturer's manuals, or even some found in books. What you have to do to keep the gun shooting well is part science, and part Voodoo, in that it can vary from day to day, based on temperature, wind and humity. It varies seasonally in most places. The good shooters are constantly changing what they do when loading and cleaning their guns to reflect the conditions on the range or in the field THAT DAY. You will find certain things that work for you any time of the year. You will settle on one thickness of cleaning patches, and a certain diameter of cleaning jag for your ramrods. I found, for instance, that when the temperature is below zero, here, that I have to clean with alcohol. Nothing else works as well. I don't try to lube the barrel other than using a pre-lubed patch for my round ball using Wonderlube. I may change that once I try running a lubed cleaning patch down the barrel after I seat the PRB and do some accuracy testing off the bench in cold weather. I found that when I lubed the whole barrel after seating my ball in the summertime, I got much more consistent velocities shot for shot, and my velocity rose about 20 fps. Now, I have to see what if anything happens below freezing. I don't like hunting when its that cold, but I have done it before, and can do it again. Learn to be flexible, but remember that the goal is to get flawless performance from your rifle, so you can concentrate on the sight picture, stance, let off and follow through, and not be worried that the gun won't go off, or that a ring of crud is building up in the barrel that will prevent you from loading the next ball. I have yet to hear from anyone who says a fouled barrel shoots to the same POI as does his unfouled, clean, barrel. My solution is to clean the barrel between shot. Some people foul their barrel before loading that first round. I never got consistency in the POI doing it that way. Your experience may differ.