Page MenuHomePhabricator

decommission bast2001
Closed, ResolvedPublicRequest

Description

This task will track the decommission-hardware of server bast2001.wikimedia.org.

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

bast2001.wikimedia.org

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

  • - disable puppet on host
  • - power down host
  • - update netbox status to Inventory (if decom) or Planned (if spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.
  • - change netbox status to offline when unracked

Related Objects

StatusSubtypeAssignedTask
ResolvedDzahn
ResolvedRequestPapaul

Event Timeline

Change 499449 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] network::constants: remove bast2001

https://gerrit.wikimedia.org/r/499449

Change 499224 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] turn bast2001 into a spare, replaced by bast2002

https://gerrit.wikimedia.org/r/499224

Change 499739 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: switch bast2001 to stretch installer

https://gerrit.wikimedia.org/r/499739

Change 499740 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] bastionhost: remove rsync and motd warning for bast2001

https://gerrit.wikimedia.org/r/499740

Change 499739 merged by Dzahn:
[operations/puppet@production] install_server: switch bast2001 to stretch installer

https://gerrit.wikimedia.org/r/499739

Dzahn changed the task status from Open to Stalled.Apr 4 2019, 12:36 PM

stalled by T219384

Change 499449 abandoned by Dzahn:
network::constants: remove bast2001

Reason:
bastions are not in constants.pp anymore since https://gerrit.wikimedia.org/r/c/operations/puppet/ /502607

https://gerrit.wikimedia.org/r/499449

Change 504095 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] remove bast2001 from bastion hosts in Hiera

https://gerrit.wikimedia.org/r/504095

Change 504095 merged by Dzahn:
[operations/puppet@production] remove bast2001 from bastion hosts in Hiera

https://gerrit.wikimedia.org/r/504095

Change 499224 merged by Dzahn:
[operations/puppet@production] turn bast2001 into a spare, replaced by bast2002

https://gerrit.wikimedia.org/r/499224

Change 499740 merged by Dzahn:
[operations/puppet@production] bastionhost: remove rsync and motd warning for bast2001

https://gerrit.wikimedia.org/r/499740

Change 504209 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: remove bast2001

https://gerrit.wikimedia.org/r/504209

Change 504209 merged by Dzahn:
[operations/puppet@production] install_server: remove bast2001

https://gerrit.wikimedia.org/r/504209

Change 504210 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove bast2001 production IPs

https://gerrit.wikimedia.org/r/504210

Dzahn changed the task status from Stalled to Open.Apr 15 2019, 9:37 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: bast2001.wikimedia.org

  • bast2001.wikimedia.org
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Dzahn updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-04-16T16:48:45Z] <mutante> puppet node clean bast2001.wikimedia.org ; puppet node deactivate bast2001.wikimedia.org ; it showed up in Icinga again despite running decom cookbook (T219492)

Mentioned in SAL (#wikimedia-operations) [2019-04-16T16:53:32Z] <mutante> bast2001 - shutdown -h now - decom'ed (T219492)

wmf-decommission-host was executed by dzahn for bast2001.wikimedia.org and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor

Change 504210 merged by Dzahn:
[operations/dns@master] remove bast2001 production IPs

https://gerrit.wikimedia.org/r/504210

Dzahn updated the task description. (Show Details)
Dzahn removed a project: Patch-For-Review.
Dzahn added a subscriber: Papaul.

Change 504381 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] remove bast2001 from site.pp

https://gerrit.wikimedia.org/r/504381

Change 504381 merged by Dzahn:
[operations/puppet@production] remove bast2001 from site.pp

https://gerrit.wikimedia.org/r/504381

papaul@asw-a-codfw# show | compare 
[edit interfaces interface-range vlan-public1-a-codfw]
-    member ge-5/0/12;
[edit interfaces interface-range disabled]
     member ge-8/0/3 { ... }
+    member ge-5/0/12;
[edit interfaces]
-   ge-5/0/12 {
-       description bast2001;
-       enable;
-   }
papaul@asw-a-codfw# run show interfaces ge-5/0/12 descriptions 
Interface       Admin Link Description
ge-5/0/12       down  down DISABLED

Change 507102 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove bast2001.mgmt

https://gerrit.wikimedia.org/r/507102

Papaul updated the task description. (Show Details)

complete

Change 507102 merged by Dzahn:
[operations/dns@master] remove bast2001.mgmt

https://gerrit.wikimedia.org/r/507102