Page MenuHomePhabricator
Create Task
Maniphest T227542

b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC)
Closed, ResolvedPublic
Actions

Description

This task will track the migration of the ps1 and ps2 to be replaced with new PDUs in rack B7-eqiad.

Each server & switch will need to have potential downtime scheduled, since this will be a live power change of the PDU towers.

These racks have a single tower for the old PDU (with and A and B side), with the new PDUs having independent A and B towers.

  • - schedule downtime for the entire list of switches and servers.
  • - before work starts, silence all icinga alerts until 8PM GMT same day
  • - Wire up one of the two towers, energize, and relocate power to it from existing/old pdu tower (now de-energized).
  • - confirm entire list of switches, routers, and servers have had their power restored from the new pdu tower
  • - Once new PDU tower is confirmed online, move on to next steps.
  • - Wire up remaining tower, energize, and relocate power to it from existing/old pdu tower (now de-energized).
  • - confirm entire list of switches, routers, and servers have had their power restored from the new pdu tower
  • - connect via serial / confirm serial connection works
  • - setup PDU following directions on https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/ServerTech#Initial_Setup
  • - update PDU model in puppet per T233129.
  • - clear icinga errors for missing ps2 input by connecting/checking connection of the rj11 cable connection between ps1 and ps2 b7-eqiad. Once it is connected, the icinga errors for the tower B infeed will clear up.

List of routers, switches, and servers

deviceroleSRE team coordinationrecommended action during maintainance
asw-b7-eqiadasw@ayounsiensure this doesn't go offline as it will take entire rack network offline
wtp1033
wtp1032
wtp1031
kafka-main1002@herronTo avoid alert noise from adjacent kafka-main hosts, schedule icinga downtime for "Kafka Broker Under Replicated Partitions" service on kafka-main100[123] as well. Perform graceful shutdown of server before maintenance, and ensure powered up when completed.
dbprov1002db provisioning/backup generation hostDBAReally nothing to do, but @jcrespo will keep an eye on it
cloudvirtan1005
cloudvirtan1004
an-worker1087@Nuria
an-worker1086@Nuria
cp1082cp systemTrafficT227542#5355289
cp1081cp systemTrafficT227542#5355289
ms-be1041ms-be systemfillipogracefully shutdown the host just before rack maintainance, and power it back online post-maintainance.
cloudvirt1022cloudvirt hostcloud-services-team@JHedden No running VMs, can happen anytime
analytics1073Analyticsfine to do any time
lvs1014lvs system@BBlackT227542#5355289
cloudvirt1020cloudvirt hostcloud-services-team@JHedden has running VMs please handle with care
druid1005Analyticsfine to do any time
ores1003
cloudnet1003cloud-services-team@JHedden is active but it has a redundant peer
restbase-dev1005
cloudcontrol1004cloud-services-team@JHedden is active but it has a redundant peer
cloudvirt1017cloudvirtcloud-services-team@JHedden has a large number of running VMs, please handle with care
mw1318mw server@Joe
mw1317mw server@Joe
mw1316mw server@Joe
mw1315mw server@Joe
mw1314mw server@Joe
mw1313mw server@Joe

Details

Due Date
Nov 15 2019, 11:00 AM
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
ps1-b7-eqiad model settingoperations/puppetproduction+4 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jul 8 2019, 10:48 PM
RobH mentioned this in T226778: Install new PDUs in rows A/B (Top level tracking task).Jul 8 2019, 10:50 PM
• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Jul 22 2019, 2:41 PM
BBlack subscribed.Jul 22 2019, 8:12 PM

lvs1014 here will need special care, Traffic should stop puppet and pybal and monitor failover to lvs1016 ahead of work, then revert afterwards. cp1081 and cp1082 here can be depooled as normal.

RobH moved this task from High Priority Task to Blocked on the ops-eqiad board.Jul 26 2019, 1:37 PM
wiki_willy renamed this task from b7-eqiad pdu refresh to b7-eqiad pdu refresh (Tuesday 11/5 @11am UTC).Aug 15 2019, 5:38 PM
RobH triaged this task as High priority.Aug 28 2019, 6:31 PM
RobH updated the task description. (Show Details)
RobH removed RobH as the assignee of this task.Aug 28 2019, 6:39 PM
RobH updated the task description. (Show Details)
RobH set Due Date to Nov 15 2019, 12:00 AM.
RobH changed Due Date from Nov 15 2019, 12:00 AM to Nov 15 2019, 11:00 AM.
RobH added subscribers: ayounsi, • Nuria, Joe.
elukey updated the task description. (Show Details)Sep 17 2019, 6:03 AM
elukey added a subscriber: herron.
RobH updated the task description. (Show Details)Oct 11 2019, 8:42 PM
• JHedden updated the task description. (Show Details)Oct 21 2019, 6:57 PM
• JHedden subscribed.
herron updated the task description. (Show Details)Oct 28 2019, 1:47 PM
wiki_willy renamed this task from b7-eqiad pdu refresh (Tuesday 11/5 @11am UTC) to b7-eqiad pdu refresh (Tuesday 11/5 @10am UTC).Oct 30 2019, 12:26 AM
wiki_willy renamed this task from b7-eqiad pdu refresh (Tuesday 11/5 @10am UTC) to b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC).Nov 4 2019, 4:19 PM
RobH updated the task description. (Show Details)Nov 4 2019, 5:26 PM
jcrespo subscribed.Nov 4 2019, 5:27 PM
Joe added a comment.Nov 4 2019, 5:27 PM

I don't want to conflict-edit the task description, but as far as the MW* and WTP* servers no action is needed.

jcrespo updated the task description. (Show Details)Nov 4 2019, 5:28 PM
wiki_willy assigned this task to • Cmjohnson.Nov 4 2019, 6:28 PM
Stashbot added a comment.Nov 5 2019, 11:59 AM

Mentioned in SAL (#wikimedia-cloud) [2019-11-05T11:59:38Z] <arturo> icinga downtime for 1h cloudcontrol1004, cloudnet1003, cloudvirt1017/1020/1022 for PDU operations in the rack T227542

Jclark-ctr subscribed.Nov 5 2019, 12:01 PM

starting pdu refresh .

Jclark-ctr added a comment.Nov 5 2019, 1:22 PM

finished pdu refresh

Jclark-ctr reassigned this task from • Cmjohnson to RobH.Nov 5 2019, 1:23 PM
Jclark-ctr updated the task description. (Show Details)
# pmshell

 1: ps1-a1-eqiad    2: ps1-a2-eqiad    3: ps1-a3-eqiad    4: ps1-a4-eqiad   
 5: ps1-a5-eqiad    6: ps1-a6-eqiad    7: ps1-a7-eqiad    8: ps1-a8-eqiad   
 9: ps1-b1-eqiad   10: ps1-b2-eqiad   11: ps1-b3-eqiad   12: ps1-b4-eqiad   
13: ps1-b5-eqiad   14: ps1-b6-eqiad   15: ps1-b7-eqiad   16: ps1-b8-eqiad   
17: asw-a1-eqiad   18: asw-a2-eqiad   19: asw-a3-eqiad   20: asw-a4-eqiad   
21: asw-a5-eqiad   22: asw-a6-eqiad   23: asw-a7-eqiad   24: asw-a8-eqiad   
25: asw-b1-eqiad   26: asw-b2-eqiad   27: asw-b3-eqiad   28: asw-b4-eqiad   
29: asw-b5-eqiad   30: asw-b6-eqiad   31: asw-b7-eqiad   32: asw-b8-eqiad   
33: re0.cr1-eqiad  34: re1.cr1-eqiad  35: re0.cr2-eqiad  36: re1.cr2-eqiad  
37: mr1-eqiad      40: msw1-eqiad     41: asw2-a5-eqiad  45: asw2-a3-eqiad  

Connect to port > 15  

Sentry Smart PDU Version 8.0n

Username:
gerritbot added a comment.Nov 5 2019, 3:50 PM

Change 548769 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] ps1-b7-eqiad model setting

https://gerrit.wikimedia.org/r/548769

gerritbot added a project: Patch-For-Review.Nov 5 2019, 3:50 PM
gerritbot added a comment.Nov 5 2019, 3:53 PM

Change 548769 merged by RobH:
[operations/puppet@production] ps1-b7-eqiad model setting

https://gerrit.wikimedia.org/r/548769

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2019, 4:10 PM
RobH reassigned this task from RobH to Jclark-ctr.Nov 5 2019, 4:45 PM
RobH updated the task description. (Show Details)
  • - clear icinga errors for missing ps2 input by connecting/checking connection of the rj11 cable connection between ps1 and ps2 b7-eqiad. Once it is connected, the icinga errors for the tower B infeed will clear up.

@Jclark-ctr: Please see the update above and address, thanks!

RobH added a comment.Nov 5 2019, 4:47 PM

Once the towers are linked, the errors on https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=ps1-b7-eqiad should clear up and go green for tower B.

Jclark-ctr closed this task as Resolved.Nov 13 2019, 4:39 PM
Jclark-ctr updated the task description. (Show Details)

confirmed link and errors cleared from icinga

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits