Page MenuHomePhabricator

rack/setup/install ps[12]-oe1[456]-esams
Open, NormalPublic

Description

This task will track the replacement of the PDUs in esams racks OE14, OE15, & OE16. All three racks have been setup for redundant power via work with Iron Mountain by @wiki_willy. Now they are ready for the installation of new PDUs ordered on T230143.

Each PDU will have to have the old pdu renamed from its current name to 'old-ps1-oe1......' Typically we rename things to their asset tag, but these PDUs pre-date asset tagging and thus have none. The new pdus will end up as the only item in netbox with the hostname ps1-rackname-site. OE14 and OE15 have to have their PS2 entries also updated, all setup items for each PDU are individually listed in the checklists below:

Please note ALL connections must be listed off on the 'cable mapping' tab of the gsheet esams elevation and cable mapping.

ps1-oe14-esams:

  • - rename the old ps1-oe14-esams to 'old-ps1-oe14-esams'
  • - once the new pdu is in place, unrack the old one and set its netbox entry to offline.
  • - receive in the new PDU from T230143 and add to netbox with the hostname ps1-oe14-esams
  • - wire up all serial/mgmt/network/power. Power will work without the PDU mgmt being configured.
  • - list off/update every single cable connection in netbox, this includes all serial/mgmt/network and POWER as these are switched PDU units and we'll have to configure remote power. This means that if an item plugs into port 1 on tower A, it should plug into port 1 on tower B. @ayounsi is familiar with this (so if uncertain please ask him.)
  • - setup PDU following directions on https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/ServerTech#Initial_Setup - @RobH can assist with this part once serial is established to the PDU.
  • - update PDU model in puppet per T233129. - @RobH can assist with this part once serial is established to the PDU.

ps2-oe14-esams:

  • - rename the old ps2-oe14-esams to 'old-ps2-oe14-esams'
  • - once the new pdu is in place, unrack the old one and set its netbox entry to offline.
  • - receive in the new PDU from T230143 and add to netbox with the hostname ps2-oe14-esams
  • - connect the link cable from ps1-oe14-esams to ps2-oe14-esams.
  • - list off/update every single cable connection in netbox, this includes all serial/mgmt/network and POWER as these are switched PDU units and we'll have to configure remote power. This means that if an item plugs into port 1 on tower A, it should plug into port 1 on tower B. @ayounsi is familiar with this (so if uncertain please ask him.)

ps1-oe15-esams:

  • - rename the old ps1-oe15-esams to 'old-ps1-oe15-esams'
  • - once the new pdu is in place, unrack the old one and set its netbox entry to offline.
  • - receive in the new PDU from T230143 and add to netbox with the hostname ps1-oe15-esams
  • - wire up all serial/mgmt/network/power. Power will work without the PDU mgmt being configured.
  • - list off/update every single cable connection in netbox, this includes all serial/mgmt/network and POWER as these are switched PDU units and we'll have to configure remote power. This means that if an item plugs into port 1 on tower A, it should plug into port 1 on tower B. @ayounsi is familiar with this (so if uncertain please ask him.)
  • - setup PDU following directions on https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/ServerTech#Initial_Setup - @RobH can assist with this part once serial is established to the PDU.
  • - update PDU model in puppet per T233129. - @RobH can assist with this part once serial is established to the PDU.

ps2-oe15-esams:

  • - OE15 did not have redundant power before, and has no existing ps2-oe15-esams to remove/decommission.
  • - receive in the new PDU from T230143 and add to netbox with the hostname ps2-oe15-esams
  • - connect the link cable from ps1-oe14-esams to ps2-oe14-esams.
  • - list off/update every single cable connection in netbox, this includes all serial/mgmt/network and POWER as these are switched PDU units and we'll have to configure remote power. This means that if an item plugs into port 1 on tower A, it should plug into port 1 on tower B. @ayounsi is familiar with this (so if uncertain please ask him.)

ps1-oe16-esams:

  • - there is no ps1-oe16-esams in netbox, so nothing to rename.
  • - receive in the new PDU from T230143 and add to netbox with the hostname ps1-oe14-esams
  • - wire up all serial/mgmt/network/power. Power will work without the PDU mgmt being configured.
  • - list off/update every single cable connection in netbox, this includes all serial/mgmt/network and POWER as these are switched PDU units and we'll have to configure remote power. This means that if an item plugs into port 1 on tower A, it should plug into port 1 on tower B. @ayounsi is familiar with this (so if uncertain please ask him.)
  • - setup PDU following directions on https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/ServerTech#Initial_Setup - @RobH can assist with this part once serial is established to the PDU.
  • - update PDU model in puppet per T233129. - @RobH can assist with this part once serial is established to the PDU.

ps2-oe16-esams:

  • - rename the old ps2-oe16-esams to 'old-ps2-oe16-esams'
  • - once the new pdu is in place, unrack the old one and set its netbox entry to offline.
  • - receive in the new PDU from T230143 and add to netbox with the hostname ps2-oe14-esams
  • - connect the link cable from ps1-oe14-esams to ps2-oe14-esams.
  • - list off/update every single cable connection in netbox, this includes all serial/mgmt/network and POWER as these are switched PDU units and we'll have to configure remote power. This means that if an item plugs into port 1 on tower A, it should plug into port 1 on tower B. @ayounsi is familiar with this (so if uncertain please ask him.)

Details

Related Gerrit Patches:
operations/puppet : productionadding esams pdus to monitoring

Event Timeline

mark triaged this task as Normal priority.Jan 3 2018, 1:13 PM
mark created this task.
RobH renamed this task from Procure and install new PDUs to rack/setup/install ps[12]-oe1[456]-esams.Tue, Oct 22, 9:34 PM
RobH claimed this task.
RobH updated the task description. (Show Details)
RobH added a parent task: Unknown Object (Task).
RobH added subscribers: wiki_willy, RobH, ayounsi.
RobH updated the task description. (Show Details)Tue, Oct 22, 9:50 PM
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)Tue, Oct 22, 9:53 PM
RobH updated the task description. (Show Details)Wed, Oct 23, 4:35 PM
RobH updated the task description. (Show Details)Thu, Oct 24, 3:13 PM

I have remotely setup both ps1-oe15-esams and ps1-oe16-esams with network configuration. They have NOT had their ports labeled, as this task doesn't list what is plugged into each port. At this time, no cable mappings have been copied to the google sheet.

Mentioned in SAL (#wikimedia-operations) [2019-10-25T15:35:35Z] <robh> ps1-oe14-esams ip info set, rebooting (wont affect servers) via T184066

RobH moved this task from Backlog to Racking Tasks on the ops-esams board.Fri, Oct 25, 3:48 PM
RobH updated the task description. (Show Details)Fri, Oct 25, 3:53 PM

Change 546674 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] adding esams pdus to monitoring

https://gerrit.wikimedia.org/r/546674

Change 546674 merged by RobH:
[operations/puppet@production] adding esams pdus to monitoring

https://gerrit.wikimedia.org/r/546674

RobH updated the task description. (Show Details)
faidon added a subscriber: faidon.Thu, Oct 31, 1:41 PM

@RobH what's the status of this?

RobH reassigned this task from RobH to Papaul.EditedThu, Oct 31, 2:31 PM
RobH added a subscriber: Papaul.

So while the cable mapping tab on this google sheet was imported into netbox, it seems that no one on-site recorded the power cord mappings?

Without this, I cannot finish the setup of the switched ports on these.

@Papaul: Are the power cable mappings written down offline somewhere?

RobH updated the task description. (Show Details)EditedThu, Oct 31, 2:57 PM

IRC Update:

@ayounsi let me know they ran out of time, and T237009 has been opened to apply labels to cords missing them.

Once all cords have labels, then they can be inputted into the PDU configuration. I've gone ahead and checked off the other boxes (after confirming their completion by logging into netbox and the PDUs directly.)

Other than the setup of the individual ports, this task is complete.

Papaul removed Papaul as the assignee of this task.Tue, Nov 5, 4:51 PM