As mentioned, tuning the lock.
IF you can hear the pan make noise, and then the rifle goes BOOM, you're ignition is too slow. In the old TV series where Daniel Boone (Fess Parker) shoots and you get a Clack-BOOM sequence..., the rifle needs work, or the sound effect was harvested from a bad gun.
Next, what position are you in when shooting? I don't shoot squirrels or any small game very often in (in fact I try to avoid) the standing-unsupported (aka "offhand") firing position. It's the toughest to hold. I either support my forward hand on a tree and then lay the rifle across the wrist of that hand, or I take a kneeling position or something "tight" as such.
Flinch..., you pull the trigger too hard standing-unsupported, or you're pulling down a bit with the forward hand expecting recoil. Often this is cured shooting from a bench a lot, and slowly slowly squeezing the trigger so that it surprises you when the rifle goes off. You need to know when the rifle is snug against your shoulder but not so tight as you are straining against the stock. If the rifle isn't properly shouldered, it bangs backwards into you, that's unpleasant, so the next shot you anticipate..and FLINCH.
You can diagnose flinch sometimes by ensuring you have an empty rifle, and then putting a hardwood chip in where the flint goes. Stand-unsupported, aim at a target, **** the rifle, and have a buddy rest a quarter on the top flat of the barrel at the muzzle behind the front sight. Squeeze off the "shot" and see if you move so much as to toss the quarter off where it's resting. You can use the middle of the barrel in a well swamped rifle, to get a level position.
Also regarding Flinch..., again the lock. The trigger pull should be crisp, not creep, but not too heavy that it makes you want to yank on the trigger instead of squeezing because your fighting the "break" of too heavy a trigger pull. A lot of reenactors have a nasty trigger flinch because they are shooting blanks from a Bess, and only blanks ever, so the heavy trigger break is a little bit more safety for them. A Pedersoli Charleville can be even harder on the trigger pull. They're not trying to hit anything, so they never realize they've gotten this habit until the day they try to shoot a ball at a target.
Spectating..., this is an "eye flinch". You're not used to seeing the movement of the lock out of the corning of your strong eye, so when you shoot, your eye comes of the front sight focus point, and looks and the dang **** where it came to rest. Of course that means you not looking at your sight and can't follow through. A lot of bench practice so you get used to the motion and can ignore it..., you have to concentrate on focusing on that front sight post to train yourself. Then move to a kneeling or seated on the ground position and work on it. I mentioned reenactors..., guys and gals that fire in line formations often pick this up with the trigger yank. They want to see if their musket is sparking well, and since again, they aren't really shooting anything other than powder, they can let their eye look at the **** striking the hammer (frizzen)
LD