Page MenuHomePhabricator
Create Task
Maniphest T185004

Decommission mw1201-mw1220
Closed, ResolvedPublic
Actions

Description

This is the last batch of very old appservers that are 2 years out of warranty. Remove them now that we've installed the new ones. This will also unblock the Row D switch upgrade that was blocked on a rebalancing of application servers.

Please note that in this range, mw1217 had been previously decommissioned, so this range only has 19 systems in it.

mw1201-1220

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1202:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1203:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1204:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1205:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1206:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1207:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1208:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1209:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1210:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1211:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1212:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1213:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1214:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1215:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1216:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1218:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1219:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

mw1220:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Removing mgmt dns mw1201-1220Cmjohnsonoperations/dnsmaster+0 -41
decom mw1201-1220 puppet repo entriesRobHoperations/puppetproduction+0 -142
decom mw1201-1220 prod dns entriesRobHoperations/dnsmaster+3 -38
site.pp: decommission mw1209-1220Giuseppe Lavagettooperations/puppetproduction+3 -14
site.pp: decommission mw1201-1208Giuseppe Lavagettooperations/puppetproduction+2 -10
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Declined• ayounsiT172459 eqiad row D switch upgrade
 Resolved• CmjohnsonT185004 Decommission mw1201-mw1220

Event Timeline

Joe created this task.Jan 16 2018, 3:30 PM
Joe triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 16 2018, 3:30 PM
Joe claimed this task.Jan 16 2018, 3:30 PM
Joe added a project: User-Joe.
Stashbot subscribed.Jan 16 2018, 4:10 PM

Mentioned in SAL (#wikimedia-operations) [2018-01-16T16:10:00Z] <_joe_> depooling mw1201-1208 from the API cluster, T185004

gerritbot added a comment.Jan 16 2018, 5:00 PM

Change 404499 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] site.pp: decommission mw1201-1208

https://gerrit.wikimedia.org/r/404499

gerritbot added a project: Patch-For-Review.Jan 16 2018, 5:00 PM

Change 404500 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] site.pp: decommission mw1209-1220

https://gerrit.wikimedia.org/r/404500

Stashbot added a comment.Jan 17 2018, 7:45 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-17T07:45:48Z] <_joe_> depooling mw1209-1220 from the appserver cluster for decommissioning, T185004

Joe added a project: hardware-requests.Jan 17 2018, 8:15 AM
Joe updated the task description. (Show Details)
gerritbot added a comment.Jan 17 2018, 8:19 AM

Change 404499 merged by Giuseppe Lavagetto:
[operations/puppet@production] site.pp: decommission mw1201-1208

https://gerrit.wikimedia.org/r/404499

gerritbot added a comment.Jan 17 2018, 8:30 AM

Change 404500 merged by Giuseppe Lavagetto:
[operations/puppet@production] site.pp: decommission mw1209-1220

https://gerrit.wikimedia.org/r/404500

Joe removed Joe as the assignee of this task.Jan 17 2018, 8:40 AM
Joe updated the task description. (Show Details)
Joe moved this task from Backlog to Blocked on others on the User-Joe board.Jan 19 2018, 10:33 AM
MoritzMuehlenhoff assigned this task to • Cmjohnson.Jan 31 2018, 2:39 PM
MoritzMuehlenhoff added a project: ops-eqiad.
MoritzMuehlenhoff removed projects: hardware-requests, HHVM.
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Feb 2 2018, 4:40 PM
RobH removed a project: Patch-For-Review.Apr 6 2018, 3:47 PM
RobH updated the task description. (Show Details)
RobH subscribed.Apr 6 2018, 4:26 PM

port info:

asw-a-eqiad:
ge-7/0/33 up up mw1204
ge-7/0/34 up up mw1205
ge-7/0/35 up up mw1206
ge-7/0/36 up up mw1207

asw-b-eqiad:
ge-7/0/32 up up mw1211
ge-7/0/33 up up mw1212
ge-7/0/34 up up mw1213
ge-7/0/35 up up mw1214
ge-7/0/36 up up mw1215
ge-8/0/34 up up mw1216
ge-8/0/36 up up mw1218
ge-8/0/37 up up mw1219
ge-8/0/38 up up mw1220

asw2-d-eqiad:
ge-5/0/1 up down mw1201
ge-5/0/2 up down mw1202
ge-5/0/3 up down mw1203
ge-5/0/4 up down mw1208
ge-5/0/5 up down mw1209
ge-5/0/6 up down mw1210

gerritbot added a comment.Apr 6 2018, 4:41 PM

Change 424623 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom mw1201-1220 prod dns entries

https://gerrit.wikimedia.org/r/424623

gerritbot added a project: Patch-For-Review.Apr 6 2018, 4:41 PM
gerritbot added a comment.Apr 6 2018, 4:43 PM

Change 424624 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom mw1201-1220 puppet repo entries

https://gerrit.wikimedia.org/r/424624

gerritbot added a comment.Apr 6 2018, 4:46 PM

Change 424623 merged by RobH:
[operations/dns@master] decom mw1201-1220 prod dns entries

https://gerrit.wikimedia.org/r/424623

gerritbot added a comment.Apr 6 2018, 4:47 PM

Change 424624 merged by RobH:
[operations/puppet@production] decom mw1201-1220 puppet repo entries

https://gerrit.wikimedia.org/r/424624

RobH updated the task description. (Show Details)EditedApr 6 2018, 4:49 PM
RobH edited projects, added: hardware-requests; removed: Patch-For-Review.

So this went unnoticed for a long time, due to the fact it lacked hardware-requests.

ALL decom/reclaim tasks should have hardware-requests as a project.

• ayounsi unsubscribed.Apr 6 2018, 4:51 PM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Apr 6 2018, 4:52 PM

These are all now ready for disk wipe.

• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.Apr 24 2018, 7:11 PM
RobH edited projects, added: decommission-hardware; removed: hardware-requests.May 3 2018, 10:57 PM
• Cmjohnson moved this task from High Priority Task to UnRacking Tasks on the ops-eqiad board.Jun 27 2018, 2:49 PM
RobH moved this task from Backlog to pending onsite steps (eqiad) on the decommission-hardware board.Jul 18 2018, 6:10 PM
gerritbot added a comment.Aug 7 2018, 5:52 PM

Change 451057 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns mw1201-1220

https://gerrit.wikimedia.org/r/451057

gerritbot added a project: Patch-For-Review.Aug 7 2018, 5:52 PM

Change 451057 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns mw1201-1220

https://gerrit.wikimedia.org/r/451057

• Cmjohnson closed this task as Resolved.Aug 7 2018, 8:06 PM
• Cmjohnson updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits