This task will track the racking, setup, and OS installation of the first 54-port PDU that we'll be using in eqiad. The PDU will be going into rack C8 on July 14, 2pm-4pm UTC, with the maintenance being performed by @Jclark-ctr. No downtime is expected.
Hostname / Racking / Installation Details
PDU upgrades are very complex. This outline/template of checklist items may require further
<ps1-c8-eqiad>:
- - apply asset tags to each tower (both primary and link towers) as well has hostname labels.
- - add new PDUs into netbox with the name prepend new- for initial netbox entry (once the old PDUs are removed and have their hostnames set to their asset tags, each PDU can be updated to remove the 'new-' prepend off the netbox hostname. example: 'new-ps1-c1-eqiad', 'new-ps2-c1-eqiad'.
The new PDUs will be mounted with one PDU per cabinet side. Also included are new offset PDU brackets. These brackets should be installed so it pushes the PDU further towards the center of the rack (to avoid the horizontal adjustment bar for the vertical rails.) Due to this, it is suggested that the 'link' pdu be installed first, leaving the 'primary' pdu for after (as these are often combined pdu towers being replaced.) This may change at the discretion of the on-sites after review.
- - list of servers in the rack are as follows (owners were notified via email on July 7):
- mw[1408-1413] - Service Ops
- icinga1001 - Observability
- db1135 - Data Persistence
- dbproxy1021 - Data Persistence
- ganeti1012 - Service Ops
- elastic1059 - Ryan Kemper
- cloudvirt[1031-1035] - WMCS new hosts
- cloudcephosd[1004-1009] - WMCS new hosts
- - check the existing PDU and all connected cables. Ensure all are properly seated and all items are receiving power from both A and B sides before continuing. Anything not seated or not receiving dual power will be rebooted by continuing this checklist.
- - install new PDU brackets for the link tower in the rack (see above note on orientation of the brackets.)
- - install link PDU into the cabinet
- - de-power old/existing B side power, and plug in new B side link PDU
- - migrate all B side power connections to new link PDU
- - Note all B side power connections, input into netbox for every single power port used.
- - When relocating power cables, please try to ensure that the A and B sides use the same port. If server bast1001 plugs into port 5 on tower B, please also have it plug into port 5 on tower A.
- - audit all B side connections to ensure all devices are receiving full power on the B side connection (any not receiving power will be rebooted when we move the A side connections next.)
- - BEFORE UNPLUGGING THE A SIDE ORIGINAL TOWER: Login to the PDU via the HTTPS interface and reset it to factory defaults!
- - Unmount existing PDU tower and set aside (if possible) to install new PDU brackets into the rack.
- - Install new PDU tower into the rack, and route power cable for easy cut-over.
- - de-power old/existing A side power, and plug in new A side link PDU
- - migrate all A side power connections to new link PDU
- - Note all A side power connections, input into netbox for every single power port used.
- - audit all A side connections to ensure all devices are receiving full power on the A side connection.