What a can of worms! There are lots of potential probems that can cause this. If you are asking for help you need to tell the gang here what you are doing. Be specific: Gun, powder, powder charge, caps, ball size, patch material, wind conditions, and sights to give the boys a start.
Larry