When I first got my Colt 2nd generation 1851 Navy in 1982, I bought a box of .375 balls with it.
Colt's instructions at the time suggest a .378" ball, but I didn't see that until later (I'd already had experience with cap and ball revolvers, but not the .36 caliber).
At the time, I still used grease over the ball.
Well, the .375 balls were nearly a slip-fit in the Colt's chambers. The balls seated fine the first time, no pull-outs. Then I slathered Crisco over the seated balls.
I fired six shots, then set to reloading again. During firing, Crisco was blown all over the cup-end of the rammer. I didn't think it would matter.
The second loading, some of those ill-fitting .375 balls got pulled out again after seating: the grease on the rammer created a suction, pulling the bullets back out with it.
Later, when I began using hard felt wads soaked in Ol' Zip Patch Grease (still sold today by Dixie Gun Works), I no longer had a problem with grease all over the rammer.
The lubricated felt wad between the ball and powder eliminated grease all over the revolver (and me).
Going to a .380" ball (cast in a Lyman mould) also solved the problem. The larger ball held tenaciously to the chamber balls, resisting movement by the rammer or recoil.
The cure for your revolver: Use a slightly larger ball. Ensure there are no burrs on the rammer face to snag the ball. Keep the rammer face free of lubricant (cotton swabs are handy for this. Dampen the swab with a bit of lighter fluid for effective grease-cutting).
But overall, I believe you'll find your problem cured if you simply use a .457" ball. Some Walker shooters have reported good results with .460" balls, but I don't know where they obtained them. Perhaps they cast them from a custom-made mould.
I wish that Hornady or Speer, or both, made swaged balls in .380 and .460 inch, for oversized .36 and .44 chambers. Seems like chambers have larger dimensions than when I first started shooting cap and ball revolvers about 1970.