Page MenuHomePhabricator

Decommission chromium and hydrogen
Closed, ResolvedPublic

Description

chromium & hydrogen have been replaced by dns100[12] - T196691

chromium:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-a-eqiad:ge-4/0/14
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

hydrogen:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-b-eqiad:ge-4/0/10
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Event Timeline

Vgutierrez added a project: Traffic.
Vgutierrez moved this task from Backlog to Hardware on the Traffic board.

Change 451360 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] site: Reinstall chromium and hydrogen as spare systems

https://gerrit.wikimedia.org/r/451360

Change 451360 merged by Vgutierrez:
[operations/puppet@production] site: Reinstall chromium and hydrogen as spare systems

https://gerrit.wikimedia.org/r/451360

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

hydrogen.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201808081600_vgutierrez_19848_hydrogen_wikimedia_org.log.

Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts:

chromium.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/201808081601_vgutierrez_20249_chromium_wikimedia_org.log.

Completed auto-reimage of hosts:

['chromium.wikimedia.org']

and were ALL successful.

Completed auto-reimage of hosts:

['hydrogen.wikimedia.org']

and were ALL successful.

Change 451789 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom of hydrogen/chromium

https://gerrit.wikimedia.org/r/451789

Change 451789 merged by RobH:
[operations/puppet@production] decom of hydrogen/chromium

https://gerrit.wikimedia.org/r/451789

Change 451790 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom of hydrogen and chromium

https://gerrit.wikimedia.org/r/451790

Change 451790 merged by RobH:
[operations/dns@master] decom of hydrogen and chromium

https://gerrit.wikimedia.org/r/451790

RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to pending onsite steps (eqiad) on the decommission-hardware board.

Change 454317 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns for decom host chromium and hydrogen

https://gerrit.wikimedia.org/r/454317

Change 454317 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns for decom host chromium and hydrogen

https://gerrit.wikimedia.org/r/454317

• Cmjohnson updated the task description. (Show Details)