This task will track the troubleshooting of [[ https://netbox.wikimedia.org/dcim/devices/1981/ | ps1-22-ulsfo ]] and [[ https://netbox.wikimedia.org/dcim/devices/1983/ | ps1-23-ulsfo ]]. These have been in alert status in icinga since the power maintenance by Digital Realty at ulsfo.
https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=1&host=ps1-22-ulsfo
https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=1&host=ps1-23-ulsfo
Investigation shows that they are delivering power to both sides, so it isn't an urgent enough issue to troubleshoot on a Friday. Instead, I'll (@robh) will go onsite next Monday and troubleshoot.
[[ https://cdn10.servertech.com/assets/documents/documents/154/original/301-9999-30_Switched_PRO2_RevE.pdf?1490979425 | Product Manual online ]]
The external reset button wipes the config, which seems worse than just reseating the NIC to powercycle the mgmt interface. However, either solution should allow for #traffic to be aware of the work in advance.
Summary of work:
* confirmed in docs that the pro2 will indeed allow hot swap of its network card (the older pro1 will not)
* scheduled work with @bblack for #traffic cooperation (no impact expected)
* unplugged all data/serial/link/temp cables from the network card (which houses the mgmt interface) and unseated it
* re-seated the NIC, repowering the mgmt interface
* plugged back in all serial/network/data and tested all connections for both ps1-22-ulsfo and ps1-23-ulsfo