Its really not that much harder to shoot fast than to shoot slow. In fact, when shooting off hand, its a bit easier, as the longer you hold, the more you shake, and the harder you struggle to pay attention to the little details..
Just dont fall into the habit of just chopping the gun upwards and jerking the trigger....you still have to go through all the phases of your shot sequence in terms of proper form: eyes locked on the target, stance set from your feet up to your knees, rifle pointed towards ground and slightly left, torso in position to meet the gun preparing for the raise...then raising by moving your left arm forward slightly as your right hand on the gun's wrist follows, at which point the left arm will begin moving the rifle upward a bit just before it finishes its final forward motion, and the right hand also follows by moving the rifle's butt upward to meet your readily positioned shoulder. With eyes never leaving the target, the sights are raised into your field of vision just as the final bit of movement locks you into good shooting form; then you squeeze a bit of the trigger and hesitant just an instant to minutely adjust your sight alignment with the target (eyes still locked on the spot you want to hit) and then squeeze that last little bit of mustard from the trigger, and boom.
Think about the fluidity of a shotgunner. They dont chuck the gun up and pull, but it appears that way. But the good ones go through all the steps one -by-one in a fluid fashion.
I think your problem would probably go away after a few sessions with this method. This is actually how I shoot a rifle myself. I just clear my head and go through the motions above (rehearsed 1000s of times at home and while shooting). Its all about motion and muscle memory. I dont use any thought or analysis once my eyes are on the target and ive decided to raise my rifle and shoot.
The only thing i can add is that if you dont have a buckhorn rear sight, then get one. Because it is a must for sight acquisition with this method. A fine rear notch throws a wrench into the fluidity your shot sequence, and creates a hiccup, and is really not any more accurate than a buckhorn either...
Go have a boatload of fun on gongs, clay pigeons, and champagne corks for practice. It gets pretty addictive.