Frustrating, isn't it? Darned right it is.
Start with a good thorough cleaning. A patent breach has more than just a fancy "hook", it has long, stout breach plug that if removed, looks like a bolt that's had the center drilled out to form a sort of powder chamber. Consequently, in order for the flash of priming to get to the powder charge it has to travel past twice as much steel tham a rifle with a regular breach and breach plug lash-up. Any oil or fouling in that chamber just makes it even more difficult.
Pay attention when you're shooting. If you get fast ignition for the first shot, but slow ignition afterward, then a dirty breech is likely contributing, if not causing, the problem.
Drilling out the vent (touch hole) will probably help some in this case too.
Pick the vent every single shot, using a pick that is thin enough and long enough to go all the way through the powder chamber and touch the far wall of the breach. When you shove in the vent pick be sure you can feel powder "crunching" under the pick. Using a patent-breach flintlock it's real easy for a short or fat vent pick to just shove fouling and gunk deeper into the vent, leaving it obstructed.
Be really careful in your loading and priming process. Do not let any priming powder block or fill the vent. Otherwise ignition will be consistently slow.
I had much the same problem with an old T/C Hawken, and found that frequent cleaning, picking the vent each and every shot, and taking special care during loading solved the problem quite nicely.
Swanny