I've got a scraper, but it's always at home - there's never any need to take either it or brushes to the range.
Scraper after only 2-3 shots? Why? A damp patch will suffice. Consider, too just a nipple pick, used in combination with a pipe cleaner thru the drum cleanout & into the bore. It doesn't take much to foul some drums and flash channels to impede cap fire into the barrel. So, step away from the scraper. You're likely breaking loose just enough powder residue to plug up the drum or nipple hole.
Also consider, after pouring in your powder charge, holding the pistol by the muzzle, and giving the barrel a couple of taps with a wooden mallet, or even just a section of big dowel or broom handle. It will settle powder, and feed a little into the drum. I use a wooden mallet for both flintlock and percussion rifles, and because of it, I never get misfires.
Nice looking pistol! I especially like the flat to round barrel. Was it a kit, or an assembly of parts you selected yourself?