For most of the rough cast surfaces, using sand paper will do the job.
Not ordinary sand paper though. The "flint" or "garnet" paper is made for sanding wood and it works rather poorly on metal.
The "Wet/Dry", black colored silicone carbide sandpaper in the rougher grades is made for use on metal and it works well. Like the name says, it can be used wetted with water or dry. For the real rough areas look for 80 or 120 grit. After the roughness is removed, switch to 180 grit, then 220 grit and finish up with 320 or 400 grit paper. You can go to finer grits but the paper finer than 1000 grit won't do much of anything. The 1000 grit and finer ones is really made for doing the final sanding on auto body paint.
I aught to also say, 3M's red, "Pro Grade Precision" works well with metal as well as wood, plastics and darn near any other sand-able material you can name. It's really good stuff. It is supposed to be used dry only.