Page MenuHomePhabricator

decommission bast2001
Closed, ResolvedPublicRequest

Description

This task will track the decommission of server bast2001.wikimedia.org.

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

bast2001.wikimedia.org

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

  • - disable puppet on host
  • - power down host
  • - update netbox status to Inventory (if decom) or Planned (if spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.
  • - change netbox status to offline when unracked

Details

Related Gerrit Patches:
operations/dns : masterremove bast2001.mgmt
operations/puppet : productionremove bast2001 from site.pp
operations/dns : masterremove bast2001 production IPs
operations/puppet : productioninstall_server: remove bast2001
operations/puppet : productionbastionhost: remove rsync and motd warning for bast2001
operations/puppet : productionturn bast2001 into a spare, replaced by bast2002
operations/puppet : productionremove bast2001 from bastion hosts in Hiera
operations/puppet : productionnetwork::constants: remove bast2001
operations/puppet : productioninstall_server: switch bast2001 to stretch installer

Related Objects

Event Timeline

Dzahn claimed this task.Mar 28 2019, 9:41 AM
Dzahn created this task.

Change 499449 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] network::constants: remove bast2001

https://gerrit.wikimedia.org/r/499449

Change 499224 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] turn bast2001 into a spare, replaced by bast2002

https://gerrit.wikimedia.org/r/499224

Change 499739 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: switch bast2001 to stretch installer

https://gerrit.wikimedia.org/r/499739

Change 499740 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] bastionhost: remove rsync and motd warning for bast2001

https://gerrit.wikimedia.org/r/499740

Change 499739 merged by Dzahn:
[operations/puppet@production] install_server: switch bast2001 to stretch installer

https://gerrit.wikimedia.org/r/499739

Dzahn changed the task status from Open to Stalled.Apr 4 2019, 12:36 PM

stalled by T219384

Change 499449 abandoned by Dzahn:
network::constants: remove bast2001

Reason:
bastions are not in constants.pp anymore since https://gerrit.wikimedia.org/r/c/operations/puppet/ /502607

https://gerrit.wikimedia.org/r/499449

Change 504095 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] remove bast2001 from bastion hosts in Hiera

https://gerrit.wikimedia.org/r/504095

Change 504095 merged by Dzahn:
[operations/puppet@production] remove bast2001 from bastion hosts in Hiera

https://gerrit.wikimedia.org/r/504095

Change 499224 merged by Dzahn:
[operations/puppet@production] turn bast2001 into a spare, replaced by bast2002

https://gerrit.wikimedia.org/r/499224

Change 499740 merged by Dzahn:
[operations/puppet@production] bastionhost: remove rsync and motd warning for bast2001

https://gerrit.wikimedia.org/r/499740

Change 504209 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: remove bast2001

https://gerrit.wikimedia.org/r/504209

Change 504209 merged by Dzahn:
[operations/puppet@production] install_server: remove bast2001

https://gerrit.wikimedia.org/r/504209

Change 504210 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove bast2001 production IPs

https://gerrit.wikimedia.org/r/504210

Dzahn changed the task status from Stalled to Open.Apr 15 2019, 9:37 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: bast2001.wikimedia.org

  • bast2001.wikimedia.org
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
Dzahn reassigned this task from Dzahn to RobH.Apr 15 2019, 11:40 PM
Dzahn updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-04-16T16:48:45Z] <mutante> puppet node clean bast2001.wikimedia.org ; puppet node deactivate bast2001.wikimedia.org ; it showed up in Icinga again despite running decom cookbook (T219492)

Mentioned in SAL (#wikimedia-operations) [2019-04-16T16:53:32Z] <mutante> bast2001 - shutdown -h now - decom'ed (T219492)

wmf-decommission-host was executed by dzahn for bast2001.wikimedia.org and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor

Change 504210 merged by Dzahn:
[operations/dns@master] remove bast2001 production IPs

https://gerrit.wikimedia.org/r/504210

Dzahn reassigned this task from RobH to Papaul.Apr 16 2019, 5:19 PM
Dzahn updated the task description. (Show Details)
Dzahn removed a project: Patch-For-Review.
Dzahn added a subscriber: Papaul.

Change 504381 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] remove bast2001 from site.pp

https://gerrit.wikimedia.org/r/504381

Change 504381 merged by Dzahn:
[operations/puppet@production] remove bast2001 from site.pp

https://gerrit.wikimedia.org/r/504381

Dzahn updated the task description. (Show Details)Apr 16 2019, 5:23 PM
Dzahn removed a project: Patch-For-Review.
papaul@asw-a-codfw# show | compare 
[edit interfaces interface-range vlan-public1-a-codfw]
-    member ge-5/0/12;
[edit interfaces interface-range disabled]
     member ge-8/0/3 { ... }
+    member ge-5/0/12;
[edit interfaces]
-   ge-5/0/12 {
-       description bast2001;
-       enable;
-   }
papaul@asw-a-codfw# run show interfaces ge-5/0/12 descriptions 
Interface       Admin Link Description
ge-5/0/12       down  down DISABLED
Papaul updated the task description. (Show Details)Apr 16 2019, 8:08 PM
RobH updated the task description. (Show Details)Apr 25 2019, 11:52 PM
Papaul updated the task description. (Show Details)Apr 29 2019, 4:25 PM

Change 507102 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove bast2001.mgmt

https://gerrit.wikimedia.org/r/507102

Papaul closed this task as Resolved.Apr 29 2019, 8:02 PM
Papaul updated the task description. (Show Details)

complete

Change 507102 merged by Dzahn:
[operations/dns@master] remove bast2001.mgmt

https://gerrit.wikimedia.org/r/507102