When you swab between shots you are pushing fouling down into the patent breech. I've had that trouble as well, I always swab between shots too.
It doesn't happen every time you swab, but the more you shoot the more it seems to start happening. Especially on a damp day when there is a lot of moisture in the air.
There are a few fixes.
First, you can just snap a cap after you swab, that will usually blow the crud back out. Works great but doubles your cap use.
Second, you can load back up again and then swab.
Third, my favorite, is find the right fit for your jag and patch combo so that the patch goes down smoothly but then bunches up and pulls the fouling out of the barrel rather than pushing it ahead of the patch. Not my idea, I read it on here: chuck your jag into a cordless drill and spin it on a flat file to turn it down. I tapered mine a bit, wanted the front a bit smaller diameter than the back. Go slow and check fit often.
I have not had a fail to fire from pushing fouling down when swabbing since doing this and I do not have to pop a cap after swabbing.
I use a 1.5" diameter cleaning patch. After turning my jag down I put some calipers on it. I have a .54 cal TC and I believe that a jag for a .52 cal would've also done the trick for me. In your case, a .54 cal cleaning jag may work about right. If you want to try that rather than turning the jag down.
I do keep a tighter fitting factory sized jag to use when I clean at the end of shooting, the turned down jag is just in my shooting bag for swabbing.