Remember how dense lead is. Anything lighter is not going to sink down into it. You have to stir in anything you use to flux. Beeswax and parafin wax are two traditional fluxing agents used in casting lead. It does not take much of either- a small grape sized lump, to flux a 20 lb. pot. It does burn and smoke, so as all casting should be done in a ventilated area, so should putting flux in the lead. The theory goes that in melting and burning off, the flux will draw some of the finer impurities to the surface, where they then can be skimmed off, as you have already done.
Most casters will first melt down " pick-up " lead and pour it into ingots. That allows them to skim off MOST of the impurities and trash that comes with any lead " Picked -up " from wherever.
When the ingots are melted, and are fluxed a second time, the rest of the trash should come off. Thereafter, you will always get an oxidize layer of off-color " skin " on the surface of the molten lead. It was my family's practice to always skim this off periodically, as we never found it contributed anything but grief to the bullets we were casting. As we cast bullets, the level of lead in the pot would drop, and when it was about a third empty, we would add another ingut. When that melted and the lead came up to temperature, we would skim the molten lead and remove the oxides with a stainless steel spoon.While this was going on , the mold was left on the top of the casting pot, on one side, to continue to be heated. Before casting again, we would also dip the mold down into the molten lead for a minute or so, or until the lead fell away from the mold easily, to make sure it was back up to casting temperature. Then we would cast another string of bullets or balls. Steel casting molds can require a lot of effort to keep them up to the right temperature. Aluminum molds are much more quickly heated, and cooled, and require a different approach. I have only very limited experience using brass molds, or brass alloy molds, but I treated them the same way I treated the Aluminum alloy molds and they seemed to cast okay. You do have to pay attention to your casting rhythm with these mold, in order to keep the mold hot enough.
I hope this helps.