I have been having the same problem.
I have been shooting black powder about 10 years now in N-SSA competition. I have always considered that I was pretty meticulous about cleaning. I typically remove the barrel and place the breech end in my utility sink with hot soapy water, and then I work the bore with a brass bristle brush and patches until the patches some out clean.
Here was the results of a test I ran a few weeks ago:
On the left is the patch from wet cleaning with soapy water in the sink.
On the right is the first dry patch down the bore.
On the top is the second dry patch down the bore.
Clearly the sink brush/patch is not getting it sufficiently clean.
I ended up getting a Chore Boy copper scrubby and cleaning my bore with it.
Patch came out dirty again:
I started to suspect leading. I used some JB deleading paste and the chore-boy again. My accuracy, which had fallen off, picked up again in the next competition.
Here is a video before/after scrubbing with the Chore boy:
Later I did a test on my M1842. Cleaned with cold water (some people had suggested flash rusting due to the hot water).
With the breech end submerged in my utility sink full of cold water, I scrubbed it with a soapy brass bristle brush, and then followed pumping action with a patch and also used my breech face scraper. The patch was clean in the water. This was my usual cleaning routine.
Then I ran 2 patches down to dry the barrel.
The 2 patches on the left are from drying:
Then I scrubbed with the Chore Boy and this was the patch after that:
I am starting to think leading is a larger problem than I thought was possible with the low velocities of muzzle loaders. I don't shoot patched bullets.
Steve