Page MenuHomePhabricator

Decommission radon
Closed, ResolvedPublic

Description

radon has been replaced by authdns1001 (T196693). It's time to decommission it

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptAug 16 2018, 8:05 AM
Vgutierrez moved this task from Triage to Hardware on the Traffic board.Aug 16 2018, 8:06 AM

Change 453099 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] authdns: Remove radon from the authdns host list

https://gerrit.wikimedia.org/r/453099

Change 453099 merged by Vgutierrez:
[operations/puppet@production] authdns: Remove radon from the authdns host list

https://gerrit.wikimedia.org/r/453099

Change 453100 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] site: Reimage radon as stretch spare system

https://gerrit.wikimedia.org/r/453100

Change 453100 merged by Vgutierrez:
[operations/puppet@production] site: Reimage radon as stretch spare system

https://gerrit.wikimedia.org/r/453100

Mentioned in SAL (#wikimedia-operations) [2018-08-16T08:35:03Z] <vgutierrez> Reimaging radon as spare system - T202040

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

radon.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201808160840_vgutierrez_4608_radon_wikimedia_org.log.

Completed auto-reimage of hosts:

['radon.wikimedia.org']

and were ALL successful.

Vgutierrez updated the task description. (Show Details)
Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Aug 16 2018, 4:05 PM
Dzahn triaged this task as Medium priority.Aug 17 2018, 6:46 PM

@Vgutierrez Not sure if this is you but before I complete the decom process for this I see these smokeping entries in puppet.

modules/smokeping/files/config.d/Targets:+++ radon
modules/smokeping/files/config.d/Targets:menu = radon
modules/smokeping/files/config.d/Targets:title = radon.wikimedia.org [C4]
modules/smokeping/files/config.d/Targets:host = radon.wikimedia.org

Change 456320 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: replace radon with dnsauth1001 as a target

https://gerrit.wikimedia.org/r/456320

Change 456320 merged by Dzahn:
[operations/puppet@production] smokeping: replace radon with cobalt as a target in C4

https://gerrit.wikimedia.org/r/456320

Dzahn added a subscriber: Dzahn.Aug 31 2018, 12:31 AM

@Cmjohnson You should be unblocked now

Mentioned in SAL (#wikimedia-operations) [2018-09-18T16:32:33Z] <mutante> radon - re-enabled disabled puppet without reason (decom) T202040

RobH assigned this task to Cmjohnson.Sep 18 2018, 9:40 PM
RobH moved this task from Backlog to pending onsite steps (eqiad) on the decommission board.
RobH added a subscriber: RobH.

Please note I did all the decom steps on the wrong task, putting them on radium task not radon. This is ready for disk wipe.

radon network port asw2-c-eqiad:ge-4/0/25

RobH updated the task description. (Show Details)
Jclark-ctr updated the task description. (Show Details)Nov 1 2019, 10:40 PM
Papaul added a subscriber: Papaul.Nov 5 2019, 9:21 PM
papaul@asw2-c-eqiad# show | compare 
[edit interfaces]
-   ge-4/0/25 {
-       description radon;
-   }
Papaul updated the task description. (Show Details)Nov 5 2019, 9:21 PM

Change 548896 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for radon

https://gerrit.wikimedia.org/r/548896

Change 548896 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for radon

https://gerrit.wikimedia.org/r/548896

Papaul closed this task as Resolved.Nov 5 2019, 9:27 PM
Papaul updated the task description. (Show Details)

Complete