In the investigation on T119631, @RobH opened a ticket to investigate temperature issues. During UnitedLayers investigation, it was discovered that one of the NICs in each cabinet's PDUs is malfunctioning. Email from UL Support quoted below:
It has come to our attention that your network management card for your
(b-side)PDU in cabinet 1.23 and (a-side) PDU in cabinet 1.22 is
malfunctioning. We are not able to remotely access and monitor your PDU
because of this issue. We would like to work with you to fix the issue sooner
than later so we do not run into any problems.
There may be some downtime with a few of your servers associated with this
work so we would like to schedule a maintenance window to your convenience so
that we can go ahead and swap out the bad PDU. If we determine that downtime
is needed, we can schedule this maintenance window during off hours so the
impact to your business continuity is at a minimum.
Please let me know if you have any questions or concerns. If you would like to
discuss this over the phone please let me know and I can accommodate this for
We will need to coordinate a downtime window. All the systems (except the mgmt switches and mr1-ulsfo) have redundant power supplies, and should remain online during this PDU replacement.
Systems without redundant power: mr1-ulsfo, msw1-ulsfo, msw2-ulsfo, altas-ulsfo, scs-ulsfo.
Downtime has been confirmed for 2016-02-17 1800-2200 GMT.