Page MenuHomePhabricator

b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC)
Open, NormalPublic

Description

This task will track the migration of the ps1 and ps2 to be replaced with new PDUs in rack B2-eqiad.

Each server & switch will need to have potential downtime scheduled, since this will be a live power change of the PDU towers.

These racks have a single tower for the old PDU (with and A and B side), with the new PDUs having independent A and B towers.

  • - schedule downtime for the entire list of switches and servers.
  • - Wire up one of the two towers, energize, and relocate power to it from existing/old pdu tower (now de-energized).
  • - confirm entire list of switches, routers, and servers have had their power restored from the new pdu tower
  • - Once new PDU tower is confirmed online, move on to next steps.
  • - Wire up remaining tower, energize, and relocate power to it from existing/old pdu tower (now de-energized).
  • - confirm entire list of switches, routers, and servers have had their power restored from the new pdu tower

List of routers, switches, and servers

deviceroleSRE team coordinationnotes
asw2-b2-eqiadasw@ayounsi
db1072dbDBAhost ready to be decommissioned - spare: T228956
ms-be1020ms-be@fgiunchedipoweroff / poweron
cloduvirt1002cloudvirtcloud-services-team
cloudvirt1001cloudvirtcloud-services-team
cloudvirt1024cloudvirtcloud-services-team
cloudvirt1018cloudvirtcloud-services-team
cloudvirt1012cloudvirtcloud-services-team
cloudvirt1009cloudvirtcloud-services-team
cloudvirt1015cloudvirtcloud-services-team
cloudvirt1008cloudvirtcloud-services-team
db1099dbDBA@jcrespo to depool this host
ms-be1047ms-be@fgiunchedipoweroff / poweron
cloudvirtan1002cloudvirtancloud-services-team
cloudvirtan1001cloudvirtancloud-services-team
an-worker1084Analyticsfine to do any time
an-worker1083Analyticsfine to do any time
cloudelastic1002cloudelasticDiscovery-Search@Gehel good to go
cp1080cpTraffic
cp1079cpTraffic
analytics1072Analyticsfine to do any time

Event Timeline

RobH created this task.Jul 8 2019, 10:44 PM
RobH updated the task description. (Show Details)Jul 10 2019, 10:49 PM
RobH added subscribers: ayounsi, fgiunchedi.
BBlack added a subscriber: BBlack.Jul 22 2019, 8:13 PM

cp1079 and cp1080 just need normal depooling process here.

From the DB side, this can be done after Thursday 25th as db1072 will no longer be a master

fgiunchedi updated the task description. (Show Details)Jul 23 2019, 9:33 AM

As of today, db1072 is no longer a master (T228243#5363931), so this rack is also good to go. db1072 will be decommissioned in a few days

Marostegui updated the task description. (Show Details)Jul 25 2019, 5:45 AM
RobH moved this task from High Priority Task to Blocked on the ops-eqiad board.Jul 26 2019, 1:37 PM
RobH removed RobH as the assignee of this task.Aug 14 2019, 4:53 PM
wiki_willy renamed this task from b2-eqiad pdu refresh to b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC).Aug 15 2019, 5:34 PM
CDanis triaged this task as Normal priority.Aug 16 2019, 1:01 PM
Marostegui updated the task description. (Show Details)Aug 19 2019, 10:38 AM
Marostegui updated the task description. (Show Details)
Marostegui added a subscriber: jcrespo.

@jcrespo - I will be on holidays this day, hence I added you on the list as primary contact for db1099 :)

Gehel updated the task description. (Show Details)Aug 19 2019, 4:18 PM
Gehel added a subscriber: Gehel.
elukey updated the task description. (Show Details)Tue, Sep 17, 5:58 AM
elukey updated the task description. (Show Details)