If you're going to enlarge the touch hole do it incrementally with wire numbered bits, rather than the English ones. There are several sizes between 3/64" and 1/16" to try first. Go with the smallest size that gives you satisfactory ignition. While it's true that a bigger TH will light the main charge faster, more gas will be escaping through it before the pressure builds up to the point that the ball is heading down the bore, thus increasing overall barrel dwell time. And it's total dwell time that is the critical number you care about the most.
Another thing to check is how profusely the frizzen sparks. While it's true that one single spark can light the whole pan, it can take a while for the whole thing to happen. If you have a profuse shower of sparks then the pan will be lit in many places simultaneously, and overall ignition will be faster.