I'll weigh in here as this is a great subject with many excellent posts to date.
If you want to see off-hand rifle shooting at its best, take a look at Olympic air rifle. This game is shot at 10 meters, off-hand, with iron sights (ok, fancy aperature sights), using a match format of 60 shots for men and 40 shots for women. The rifles are all .177 pellet rifles, most are using compressed air (at 3000 PSI) as a propellant. The target is a 10-ring target where the 10-ring itself is a dot smaller than the period at the end of this sentence. And if you're not hitting that dot on at least 45 of the 60 shots, you ain't even competitive. The good shooters hit it at least 50 out of 60 shots, and all their other shots never go outside the 9-ring (which happens to be just about .177" in diameter!). These shooters use specialized pants, boots, jackets and gloves to achieve very little body movement. But they still have to support their 10+lb rifles for the shot, and their margin for error is so small it makes MOA groups look like shootgun patterns.
In this game mental control is what makes you or breaks you. Breathing, relaxation, trigger control, and stance are all important factors. But the single biggest factor remains concentration. If your brain is not 100% focused on shooting a "10" then you'll blow the shot, guaranteed.
One additional point here... One factor in offhand shooting which is often overlooked is balance. If we don't have good balance and fast recovery from external stimulus (like wind blowing our bodies around) then we will shoot poorly. Trust me, I know about this as I have terrible balance! A training technique that I have found to be helpful in developing balance is the use of a BIM board. BIM = Balance Improvement Machine.
To make one, take a sheet of plywood (I used 3/4")and cut from it a 24" diameter circle. At the centre of the circle affix with three or for screws a wooden ball or a piece of wood with a radius turned on one end of it and a flat on the other. The flat spot mates with the plywood to give good bearing surface for your screws to take hold on. I used a 4" pressure-treated wooden ball from Home Depot. It was originally designed for topping off a deck post - just cut off the portion of the post topper that contains the lag screw so you get a ball that's flat on one side. Once you've got this put together, set it on a hard surface with the ball against the ground (don't do this on hardwood or vinyl floors as you WILL leave a dent in your floor!). Now try to stand on the piece of plywood. I wish you luck. Eventually, with lots of work, you'll be able to stand upright on this contraption. And once you can, you'll see our group sizes shrink. This has certainly been the case with my pistol groups.
And just as an aside, I know one pistol shooter who actually is able to not only stand upright on this BIM board with gyrating all over the place, but he's put so much work into developing his balance and coordination that he can actually shoot from this position! Needless to say, he's quite an accomplished athlete who competes internatinally.
