I've heard that cleaning the barrel will throw off your shot, is that true?
Yep... if by that you mean that the first shot through a clean, cold barrel will likely be in a different spot than one from a fouled barrel, or a warm one.
A perfect example is my 20 ga smoothbore. The first shot from a clean cold barrel will take the center right out of the bull's eye. By the fifth shot, she is putting them a little high and to the right. Don't ask me why there, but she is consistent.
What do I do about it? I leave the sight just like it is. If I'm hunting with round ball, say, for deer, the shot that counts will be the first from a clean, cold barrel, so I want her sighted in for that POI. If I'm punching targets, I aim center hold for the first few shots, then hold a little low and to the left for all that follow.
To get your rifle sighted in for that "clean, cold" shot, you will have to swab between shots as well as give the barrel time to cool off.
I suppose a fellow that did only target shooting would sight her in for the POI that results from a fouled warm barrel, and just start the day with a few "fouling shots" to get her shooting so POA and POI were lined up. A fellow that had a rifle sighted in that way would probably start a day of hunting by firing off a few shots to get his barrel fouled, then move to his hunting area.
Firing off black powder leaves behind a salty, corrosive residue that will rust out your barrel if it is not cleaned out. I clean my MLs as soon as I finish for the day, and sometimes during a session if the fouling makes it hard to load. Flushing the barrel out with water (plugging the vent, filling the barrel with water, then pouring it back out) is the way you get to that area at the breech end of the barrel, followed by swabbing with a patch, etc. See the links Roundball recommended for more info.