Could be you, could be the bullets, could be the lube, could be the gun. Fliers are generally not the rifle. Are the fliers consistantly to one side or above/below. I have a chart at home to diagnose 'consistant' fliers.
You: breaking concentration - or concentrating differently. Some folks have a tremendous focus on the target, and it actually tires them out so that it falls apart at some point. Or, if you relax normally and for one shot you are tensed up you might jerk the trigger. Inconsistant anything will pull a flier. Follow through - take a breath after the shot before you move at all.
Bullets: If you are using cast bullets weigh 20 of them. Are they the same weight? Voids in the casting, or fluxes of odd metals in the lead, will make the ball fly erratically.
Lube: Are you getting too much or not enough on the patches? A really sloppy patch will cause the shot to print differently if all the others were thinly lubed. It's like shooting a heavier ball (and can soak into the powder, behaving like a lighter loading) A thinly lubed patch can burn through, tear, or leave uneven fouling. Is the fouling building up to the point where it hampers the ball? Switch lubes - especially from liquid to grease or vice versa and see if it helps.
Too heavy/too light or inconsistant trigger pull. I've been told that a target rifle should always surprise you when it goes off. Well, maybe. But it shouldn't go off before you're prepared or 'zoned in.' A gritty or poorly degigned trigger that is inconsistant can cause a flier, too.