Thanks to librenms alerting (see related T224180 on getting these sent to dc ops), we are able to see that the humidity in our codfw facility is quite high.

https://librenms.wikimedia.org/devices/type=power/format=graph_humidity/from=-24h/to=now/

If you compare codfw to other sites, its very high. Most best practices peg datacenter humidity between 45-55%, as less can cause spark hazards and more cause condensation issues.

Please note this is NOT something that @Papaul would have been alerted to, not until after notification for alerts is sorted out on T224180.)

CyrusOne support will need to be contacted about getting the humidity lowered within our datahall/cage. This should not result in any charges (but we should make sure they clarify that during our request) as power/temp/humidity controls are part of our lease.