I have only been building guns for 42 years, and seen more than one nipple blown out from people removing them every time. Remove them once a year to inspect for wear and re-lube, or to replace if accuracy falls off.Always remove the nipple!
Taking them out to clean? You reduce hydraulic pressure, making it more difficult to clean the breech, especially on patent breeches.