Skeeter Skelton talks about the quick, almost "throwing" motion Cavalry vets described to shoot from horseback because it gave you a second of stability to aim a snap shot on a galloping horse .
Honestly if I were a Cavalryman fighting other cavalry I'd shoot at the horse, a much bigger target than trying to pick a rider off. I'm pretty certain I can hit a horse size target at 100 without even using sights. I'm sure the odds are 0 any of us will ever or have done this so all we can do is read and talk about it.
How many documented battles were there in the Civil War that actually involved mounted Cavalrymen shooting at each other? Probably not many. Or Cavalry charges into Infantry? Probably very few.
They were used as reconnaisance and as fast moving "shock troops" that could dismount and like was said, use carbines or whatever weapon they carried, in the case of some Confederate cavalry, a full length rifle-musket if nothing else could be issued.
As a high mediocre pistol shot who just enjoys shooting, if I can hit a B-27 Silhouette at 100 yards with a .44 percussion revolver , standing with one hand than anyone can do it with a little bit of practice. It's not like you need to be an expert. Put it in your sights and squeeze, it will hit somewhere on the target. At 50, hold at the belt or belly, you'll hit it. This is the whole design of the sights. I'm not sure why people feel this is a "flaw" when even original revolvers hit high at 25.