Page MenuHomePhabricator
Create Task
Maniphest T169518

Decommission esams ms-fe / ms-be
Closed, ResolvedPublic
Actions

Description

The esams machines used for swift are old and unused for production purposes, we should decom

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt Dns for ms-be300[1-4] and ms-fe300[1-2]operations/dnsmaster+1 -12
decom esams swift machines, keep mgmtoperations/dnsmaster+0 -13
decom esams swift machines, rm from puppet/dhcpoperations/puppetproduction+0 -48
Decom ms-fe.svc.esams.wmnetoperations/dnsmaster+0 -4
Decom swift cluster in esamsoperations/puppetproduction+2 -61
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Jul 3 2017, 10:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2017, 10:59 AM
gerritbot subscribed.Jul 3 2017, 11:00 AM

Change 362965 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Decom swift cluster in esams

https://gerrit.wikimedia.org/r/362965

gerritbot added a project: Patch-For-Review.Jul 3 2017, 11:00 AM
gerritbot added a comment.Jul 3 2017, 11:05 AM

Change 362965 merged by Filippo Giunchedi:
[operations/puppet@production] Decom swift cluster in esams

https://gerrit.wikimedia.org/r/362965

gerritbot added a comment.Jul 3 2017, 11:12 AM

Change 362968 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] Decom ms-fe.svc.esams.wmnet

https://gerrit.wikimedia.org/r/362968

gerritbot added a comment.Jul 3 2017, 11:14 AM

Change 362968 merged by Filippo Giunchedi:
[operations/dns@master] Decom ms-fe.svc.esams.wmnet

https://gerrit.wikimedia.org/r/362968

fgiunchedi mentioned this in T91637: Rack ms-be3006 and ms-be3007.Jul 3 2017, 11:17 AM
fgiunchedi mentioned this in T86784: setup the 2 new esams ms-be systems.
fgiunchedi mentioned this in T83811: ms-be3003 sdk (bay 11) broken.
fgiunchedi added a project: User-fgiunchedi.Jul 4 2017, 12:47 PM
fgiunchedi moved this task from Backlog to Radar on the User-fgiunchedi board.
fgiunchedi added a comment.Jul 12 2017, 7:50 AM

I've reimaged all ms-be / ms-fe in esams and wiped data disks on the former, left to do is to wipe only the OS disks when the time comes for decom

fgiunchedi triaged this task as Medium priority.Jul 19 2017, 10:09 AM
faidon moved this task from Backlog to Decommission on the ops-esams board.Aug 29 2017, 3:05 PM
mark added a parent task: T184063: Remove all decommissioned hardware.Jan 3 2018, 1:04 PM
Dzahn updated the task description. (Show Details)Jan 9 2018, 6:03 PM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Jan 9 2018, 6:10 PM

Change 403215 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] decom esams swift machines, rm from puppet/dhcp

https://gerrit.wikimedia.org/r/403215

gerritbot added a comment.Jan 9 2018, 6:14 PM

Change 403216 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] decom esams swift machines, keep mgmt

https://gerrit.wikimedia.org/r/403216

Dzahn updated the task description. (Show Details)Jan 9 2018, 6:21 PM
gerritbot added a comment.Jan 9 2018, 6:22 PM

Change 403215 merged by Dzahn:
[operations/puppet@production] decom esams swift machines, rm from puppet/dhcp

https://gerrit.wikimedia.org/r/403215

Stashbot subscribed.Jan 9 2018, 6:42 PM

Mentioned in SAL (#wikimedia-operations) [2018-01-09T18:42:53Z] <mutante> ms-fe3002,ms-fe3001 - powering down, removing from puppet and icinga, ms-be* removing from puppet/icinga (T169518)

gerritbot added a comment.Jan 9 2018, 6:56 PM

Change 403216 merged by Dzahn:
[operations/dns@master] decom esams swift machines, keep mgmt

https://gerrit.wikimedia.org/r/403216

Dzahn updated the task description. (Show Details)Jan 9 2018, 6:57 PM
Dzahn assigned this task to mark.Jan 9 2018, 7:10 PM
Dzahn added subscribers: mark, Dzahn.

@mark @fgiunchedi They are shutdown and removed from Icinga and DNS now. Only the "disable switch port" part i could not do due to lack of access. I copied the check boxes from the decom template on the wikitech server lifecycle page.

fgiunchedi added a comment.Jan 10 2018, 9:01 AM

Thanks a lot @Dzahn for taking care of this!

fgiunchedi awarded a token.Jan 10 2018, 9:01 AM
fgiunchedi rescinded a token.
fgiunchedi awarded a token.Jan 10 2018, 1:34 PM
fgiunchedi removed projects: User-fgiunchedi, Patch-For-Review.Mar 16 2018, 11:43 AM
fgiunchedi added a project: hardware-requests.Mar 16 2018, 11:55 AM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Apr 11 2018, 5:49 PM
RobH edited projects, added decommission-hardware; removed hardware-requests.May 3 2018, 10:41 PM
RobH moved this task from Backlog to pending onsite steps (esams) on the decommission-hardware board.Jul 18 2018, 6:11 PM
Dzahn unsubscribed.Jul 24 2018, 9:09 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:20 PM
Papaul claimed this task.Oct 17 2019, 12:35 AM
Papaul updated the task description. (Show Details)Oct 27 2019, 7:07 PM
gerritbot added a comment.Oct 30 2019, 10:51 PM

Change 547337 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt Dns for ms-be300[1-4] and ms-fe300[1-2]

https://gerrit.wikimedia.org/r/547337

gerritbot added a project: Patch-For-Review.Oct 30 2019, 10:51 PM
gerritbot added a comment.Oct 30 2019, 10:56 PM

Change 547337 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt Dns for ms-be300[1-4] and ms-fe300[1-2]

https://gerrit.wikimedia.org/r/547337

Papaul closed this task as Resolved.Oct 30 2019, 10:57 PM
Papaul updated the task description. (Show Details)

Complete

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL