Page MenuHomePhabricator
Create Task
Maniphest T202040

Decommission radon
Closed, ResolvedPublic
Actions

Description

radon has been replaced by authdns1001 (T196693). It's time to decommission it

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for radonoperations/dnsmaster+0 -4
smokeping: replace radon with cobalt as a target in C4operations/puppetproduction+4 -4
site: Reimage radon as stretch spare systemoperations/puppetproduction+2 -4
authdns: Remove radon from the authdns host listoperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects

Event Timeline

Vgutierrez created this task.Aug 16 2018, 8:05 AM
Restricted Application added a project: SRE. · View Herald TranscriptAug 16 2018, 8:05 AM
Vgutierrez moved this task from Backlog to Hardware on the Traffic board.Aug 16 2018, 8:06 AM
gerritbot subscribed.Aug 16 2018, 8:08 AM

Change 453099 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] authdns: Remove radon from the authdns host list

https://gerrit.wikimedia.org/r/453099

gerritbot added a project: Patch-For-Review.Aug 16 2018, 8:08 AM
gerritbot added a comment.Aug 16 2018, 8:15 AM

Change 453099 merged by Vgutierrez:
[operations/puppet@production] authdns: Remove radon from the authdns host list

https://gerrit.wikimedia.org/r/453099

gerritbot added a comment.Aug 16 2018, 8:23 AM

Change 453100 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] site: Reimage radon as stretch spare system

https://gerrit.wikimedia.org/r/453100

gerritbot added a comment.Aug 16 2018, 8:34 AM

Change 453100 merged by Vgutierrez:
[operations/puppet@production] site: Reimage radon as stretch spare system

https://gerrit.wikimedia.org/r/453100

Stashbot subscribed.Aug 16 2018, 8:35 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-16T08:35:03Z] <vgutierrez> Reimaging radon as spare system - T202040

ops-monitoring-bot subscribed.Aug 16 2018, 8:40 AM

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

radon.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201808160840_vgutierrez_4608_radon_wikimedia_org.log.

ops-monitoring-bot added a comment.Aug 16 2018, 9:19 AM

Completed auto-reimage of hosts:

['radon.wikimedia.org']

and were ALL successful.

Vgutierrez removed a project: Patch-For-Review.Aug 16 2018, 9:23 AM
Vgutierrez updated the task description. (Show Details)
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Aug 16 2018, 4:05 PM
Dzahn triaged this task as Medium priority.Aug 17 2018, 6:46 PM
• Cmjohnson subscribed.Aug 29 2018, 6:31 PM

@Vgutierrez Not sure if this is you but before I complete the decom process for this I see these smokeping entries in puppet.

modules/smokeping/files/config.d/Targets:+++ radon
modules/smokeping/files/config.d/Targets:menu = radon
modules/smokeping/files/config.d/Targets:title = radon.wikimedia.org [C4]
modules/smokeping/files/config.d/Targets:host = radon.wikimedia.org

gerritbot added a comment.Aug 30 2018, 1:20 AM

Change 456320 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: replace radon with dnsauth1001 as a target

https://gerrit.wikimedia.org/r/456320

gerritbot added a project: Patch-For-Review.Aug 30 2018, 1:20 AM
gerritbot added a comment.Aug 31 2018, 12:29 AM

Change 456320 merged by Dzahn:
[operations/puppet@production] smokeping: replace radon with cobalt as a target in C4

https://gerrit.wikimedia.org/r/456320

Dzahn subscribed.Aug 31 2018, 12:31 AM

@Cmjohnson You should be unblocked now

Stashbot added a comment.Sep 18 2018, 4:32 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-18T16:32:33Z] <mutante> radon - re-enabled disabled puppet without reason (decom) T202040

Dzahn mentioned this in T203861: decom radium.Sep 18 2018, 9:39 PM
RobH assigned this task to • Cmjohnson.Sep 18 2018, 9:40 PM
RobH moved this task from Backlog to pending onsite steps (eqiad) on the decommission-hardware board.
RobH subscribed.

Please note I did all the decom steps on the wrong task, putting them on radium task not radon. This is ready for disk wipe.

In T203861#4595920, @RobH wrote:

radon network port asw2-c-eqiad:ge-4/0/25

RobH removed a project: Patch-For-Review.Sep 18 2018, 9:42 PM
RobH updated the task description. (Show Details)
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:16 PM
Jclark-ctr claimed this task.Oct 15 2019, 9:05 PM
Jclark-ctr updated the task description. (Show Details)Nov 1 2019, 10:40 PM
Papaul subscribed.Nov 5 2019, 9:21 PM
papaul@asw2-c-eqiad# show | compare 
[edit interfaces]
-   ge-4/0/25 {
-       description radon;
-   }
Papaul updated the task description. (Show Details)Nov 5 2019, 9:21 PM
gerritbot added a comment.Nov 5 2019, 9:24 PM

Change 548896 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for radon

https://gerrit.wikimedia.org/r/548896

gerritbot added a project: Patch-For-Review.Nov 5 2019, 9:24 PM

Change 548896 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for radon

https://gerrit.wikimedia.org/r/548896

Papaul closed this task as Resolved.Nov 5 2019, 9:27 PM
Papaul updated the task description. (Show Details)

Complete

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2019, 10:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL