Your swabbing method may or may not be a part of the problem with the mis-fires.
If you are shoving the dampened patch down the bore and then pumping the ramrod up and down to scrape off the fouling before removing it, you are causing loose fouling to blow back into the flame channel that runs from the bottom of the nipple to the bore. That can cause a lot of mis-fires.
The right way to swab IMO, is to dampen the patch and shove it all the way to the breech in one smooth motion. Then, let it sit there for 3-5 seconds to allow the alcohol or water to soak into the fouling on the bore. After this wait, pull the ramrod and wet patch back out of the bore in a slow, smooth manner.
The wet fouling will be wiped out of the bore without loose fouling getting blown back into the flame channel.
You can follow this with a dry patch to dry off any dampness that remains in the bore but IMO it really isn't necessary.
If you haven't done it, buy some pipe cleaners. The kind pipe smokers use to clean the pipe stem.
When you clean your rifle, wet a pipe cleaner and run it thru the flame channel under the nipple. You should be able to see it coming into the bore if you peek down it with a flashlight.
Then, remove the pipe cleaner and run a clean, dry one back and forth in the flame channel.
Another tip. When you loading the rifle make sure there is no cap on the nipple and the hammer is at half cock. This allows air to be blown out thru the nipple as the patched ball or the bullet is rammed down the bore. The venting air will help to carry fresh powder thru the flame channel to the bottom of the nipple.
Have fun.