Page MenuHomePhabricator
Create Task
Maniphest T220504

Decommission sarin
Closed, ResolvedPublic
Actions

Description

sarin has been replaced by cumin2001. Before this host can be removed, the MySQL grants need to be removed:
https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/466833/

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host

[x ] - disable switch port

  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for sarin,db2050 and db2055operations/dnsmaster+0 -12
mysql: remove grants for sarin and neodymiumoperations/puppetproduction+0 -0
Remove DNS entries for neodymium/sarinoperations/dnsmaster+0 -8
Remove site.pp entries for neodymium/sarinoperations/puppetproduction+0 -8
Decomission sarinoperations/puppetproduction+0 -8
Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.Apr 9 2019, 1:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 9 2019, 1:45 PM
MoritzMuehlenhoff triaged this task as Medium priority.Apr 9 2019, 1:47 PM
MoritzMuehlenhoff added projects: ops-codfw, decommission-hardware.
Restricted Application added a project: SRE. · View Herald TranscriptApr 9 2019, 1:47 PM
Dzahn moved this task from Backlog to Decommission on the ops-codfw board.Apr 12 2019, 12:07 AM
RobH moved this task from Backlog to Blocked on Service Owners on the decommission-hardware board.Apr 23 2019, 7:38 PM
MoritzMuehlenhoff mentioned this in T224549: Track remaining jessie systems in production.May 29 2019, 10:18 AM
jbond subscribed.Jun 6 2019, 6:56 PM

going to re-image this server to stretch, testing changes to late_command.sh

ops-monitoring-bot added a comment.Jun 6 2019, 6:58 PM

Script wmf-auto-reimage was launched by jbond on cumin1001.eqiad.wmnet for hosts:

['sarin.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906061858_jbond_159535.log.

ops-monitoring-bot added a comment.Jun 6 2019, 8:05 PM

Completed auto-reimage of hosts:

['sarin.codfw.wmnet']

Of which those FAILED:

['sarin.codfw.wmnet']
ops-monitoring-bot added a comment.Jun 7 2019, 10:03 AM

Script wmf-auto-reimage was launched by jbond on cumin1001.eqiad.wmnet for hosts:

['sarin.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201906071003_jbond_74120.log.

ops-monitoring-bot added a comment.Jun 7 2019, 11:01 AM

Completed auto-reimage of hosts:

['sarin.codfw.wmnet']

Of which those FAILED:

['sarin.codfw.wmnet']
RobH assigned this task to MoritzMuehlenhoff.Jul 26 2019, 3:29 PM
RobH subscribed.
In T220503#5369016, @RobH wrote:

@MoritzMuehlenhoff,

Have the grants for this system been removed so we can move forward with decommission? Directed this to you since you created this task, and I assume you are a service owner?

Please comment and if its ready to start the decom process, check off the boxes and assign to me for followup. Thanks in advance!

gerritbot added a comment.Aug 1 2019, 10:24 AM

Change 527043 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] mysql: remove grants for sarin and neodymium

https://gerrit.wikimedia.org/r/527043

gerritbot added a project: Patch-For-Review.Aug 1 2019, 10:24 AM
Marostegui mentioned this in T229796: Remove sarin and neodymium GRANTs from all the databases.Aug 5 2019, 10:22 AM
Marostegui added a parent task: T229796: Remove sarin and neodymium GRANTs from all the databases.
Marostegui removed a parent task: T229796: Remove sarin and neodymium GRANTs from all the databases.
Marostegui added a subtask: T229796: Remove sarin and neodymium GRANTs from all the databases.Aug 5 2019, 10:24 AM
MoritzMuehlenhoff added a comment.Aug 5 2019, 10:30 AM

@RobH This needs to wait until https://phabricator.wikimedia.org/T229796 is complete, I'll reassign the bug to you when that's done.

Marostegui closed subtask T229796: Remove sarin and neodymium GRANTs from all the databases as Resolved.Aug 28 2019, 6:21 AM
gerritbot added a comment.Sep 5 2019, 10:18 AM

Change 534598 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Decomission sarin

https://gerrit.wikimedia.org/r/534598

gerritbot added a comment.Sep 5 2019, 10:21 AM

Change 534598 merged by Muehlenhoff:
[operations/puppet@production] Decomission sarin

https://gerrit.wikimedia.org/r/534598

MoritzMuehlenhoff reassigned this task from MoritzMuehlenhoff to RobH.Sep 5 2019, 10:21 AM
MoritzMuehlenhoff updated the task description. (Show Details)

This is ready to be decommissioned now.

MoritzMuehlenhoff moved this task from Blocked on Service Owners to Ready for Decommission on the decommission-hardware board.Sep 5 2019, 11:55 AM
MoritzMuehlenhoff claimed this task.Sep 20 2019, 7:39 AM
ops-monitoring-bot added a comment.Sep 20 2019, 7:46 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: sarin.codfw.wmnet

  • sarin.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Sep 20 2019, 7:49 AM

Change 538159 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove site.pp entries for neodymium/sarin

https://gerrit.wikimedia.org/r/538159

gerritbot added a comment.Sep 20 2019, 7:49 AM

Change 538159 merged by Muehlenhoff:
[operations/puppet@production] Remove site.pp entries for neodymium/sarin

https://gerrit.wikimedia.org/r/538159

gerritbot added a comment.Sep 20 2019, 7:55 AM

Change 538160 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Remove DNS entries for neodymium/sarin

https://gerrit.wikimedia.org/r/538160

gerritbot added a comment.Sep 20 2019, 7:56 AM

Change 538160 merged by Muehlenhoff:
[operations/dns@master] Remove DNS entries for neodymium/sarin

https://gerrit.wikimedia.org/r/538160

MoritzMuehlenhoff reassigned this task from MoritzMuehlenhoff to RobH.Sep 20 2019, 8:00 AM
MoritzMuehlenhoff updated the task description. (Show Details)
MoritzMuehlenhoff reassigned this task from RobH to Papaul.Sep 25 2019, 6:30 PM
MoritzMuehlenhoff moved this task from Ready for Decommission to pending onsite steps (eqiad) on the decommission-hardware board.
Papaul added a comment.Sep 25 2019, 6:40 PM
papaul@asw-a-codfw# show | compare 
[edit interfaces interface-range vlan-private1-a-codfw]
-    member ge-5/0/16;
[edit interfaces interface-range disabled]
     member ge-6/0/15 { ... }
+    member ge-5/0/16;
[edit interfaces]
-   ge-5/0/16 {
-       description sarin;
-       enable;
-   }
Papaul updated the task description. (Show Details)Sep 25 2019, 6:40 PM
gerritbot added a comment.Sep 26 2019, 4:23 AM

Change 527043 abandoned by Jcrespo:
mysql: remove grants for sarin and neodymium

Reason:
Done somewhere else

https://gerrit.wikimedia.org/r/527043

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 5:11 AM
Papaul updated the task description. (Show Details)Oct 7 2019, 3:16 PM
Papaul updated the task description. (Show Details)Oct 7 2019, 8:06 PM
gerritbot added a comment.Oct 9 2019, 7:58 PM

Change 541907 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for sarin,db2050 and db2055

https://gerrit.wikimedia.org/r/541907

gerritbot added a project: Patch-For-Review.Oct 9 2019, 7:58 PM
gerritbot added a comment.Oct 9 2019, 8:17 PM

Change 541907 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for sarin,db2050 and db2055

https://gerrit.wikimedia.org/r/541907

Papaul closed this task as Resolved.Oct 9 2019, 8:20 PM
Papaul updated the task description. (Show Details)

complete

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2019, 9:11 PM
RobH unsubscribed.Oct 9 2019, 9:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL