Sounds to me like a lead problem.. you need to use the purest lead you can find as this is the softest. Be very wary of using range scrap as most breech loading lead bullets are alloy with added tin. With a muzzle loading revolver, you are looking for a rim of lead being shaved off as the ball goes into the cylinder. This needs a sharp edged cylinder with NO chamfer. You should not need to add any further leverage to the rammer, If you do then the lead is too hard! *
Whatever you do, don't ever ream a gun unless you really know what you are doing.. you are just creating 100+ years of wear at a single stroke. There is a zero chance that "the gun is wrong" it is either you or the bullet size. NEVER remove metal from a gun unless you are 100% sure it should not be there, and even then it is better to hammer the metal back to where it was rather than cut it off.. Dremel is responsible for more gun history lost than any war or politician!
* On second thoughts, just go ahead.. I need the income from replacing and repairing broken rammers!