Page MenuHomePhabricator

Decommission mw1170-mw1179
Closed, ResolvedPublic

Description

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - replace with role::spare::system

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update racktables with result
  • - switch port configration removed from switch once system is unracked.
  • - mgmt dns entries removed.

Details

Related Gerrit Patches:
operations/dns : mastermw117[0-9] decommission
operations/puppet : productiondecom mw117[0-9]
operations/puppet : productionrole::mediawiki::appservers: move mw1170-1179 to role::spare::system

Event Timeline

Joe created this task.Jun 19 2017, 11:00 AM

Mentioned in SAL (#wikimedia-operations) [2017-06-19T11:01:15Z] <_joe_> depooling mw1170-mw1179 for decommissioning, T168271

Change 359920 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] role::mediawiki::appservers: move mw1170-1179 to role::spare::system

https://gerrit.wikimedia.org/r/359920

Change 359920 merged by Giuseppe Lavagetto:
[operations/puppet@production] role::mediawiki::appservers: move mw1170-1179 to role::spare::system

https://gerrit.wikimedia.org/r/359920

Joe updated the task description. (Show Details)Jun 19 2017, 12:47 PM
Joe updated the task description. (Show Details)Jun 19 2017, 12:49 PM

@Cmjohnson please proceed to decom/derack these servers and rack new ones in their place.

Mentioned in SAL (#wikimedia-operations) [2017-06-26T21:50:12Z] <robh> shutting down and decommissioning mw117[0-9] per T168271

RobH updated the task description. (Show Details)Jun 26 2017, 9:55 PM
RobH added a subscriber: RobH.

ge-6/0/9 up up mw1170
ge-6/0/10 up up mw1171
ge-6/0/11 up up mw1172
ge-6/0/12 up up mw1173
ge-6/0/13 up up mw1174
ge-6/0/14 up up mw1175
ge-6/0/15 up up mw1176
ge-6/0/16 up up mw1177
ge-6/0/17 up up mw1178
ge-6/0/18 up down mw1179

Change 361581 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom mw117[0-9]

https://gerrit.wikimedia.org/r/361581

Change 361582 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] mw117[0-9] decommission

https://gerrit.wikimedia.org/r/361582

Change 361581 merged by RobH:
[operations/puppet@production] decom mw117[0-9]

https://gerrit.wikimedia.org/r/361581

Change 361582 merged by RobH:
[operations/dns@master] mw117[0-9] decommission

https://gerrit.wikimedia.org/r/361582

RobH assigned this task to Cmjohnson.Jun 26 2017, 10:07 PM
RobH updated the task description. (Show Details)

All non-onsite steps have been completed, and these hosts now await disk wipes.

RobH moved this task from Backlog to Not urgent on the ops-eqiad board.Jun 26 2017, 10:08 PM
Joe moved this task from Backlog to Blocked on others on the User-Joe board.Jul 3 2017, 7:55 AM
Cmjohnson moved this task from Not urgent to Decommission on the ops-eqiad board.Jul 20 2017, 3:24 PM
Joe raised the priority of this task from Medium to High.Aug 28 2017, 7:52 AM
Cmjohnson closed this task as Resolved.Sep 21 2017, 4:20 PM
Cmjohnson updated the task description. (Show Details)

all steps have been completed.