Page MenuHomePhabricator

decommission bast3002
Closed, ResolvedPublicRequest

Description

This task will track the decommission of server bast3002.wikimedia.org (CURRENTLY ACTIVE SERVER, only after bast3004 has been setup)

With the launch of updates to the decom cookbook, the majority of these steps can be handled by the service owners directly. The DC Ops team only gets involved once the system has been fully removed from service and powered down by the decommission cookbook.

bast3002.wikimedia.org

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal.
  • - remove all remaining puppet references (include role::spare) and all host entries in the puppet repo
  • - remove ALL dns entries except the asset tag mgmt entries.
  • - reassign task from service owner to DC ops team member depending on site of server: codfw = @Papaul, eqiad = @Jclark-ctr, all other sites = @RobH.

End service owner steps / Begin DC-Ops team steps:

  • - disable switch port / set to asset tag if host isn't being unracked / remove from switch if being unracked.
  • - system disks wiped (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned. If uncertain, ask @wiki_willy.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Gerrit Patches:
operations/dns : masterDNS: Remove mgmt DNS for bast3002
operations/puppet : productionsite: replace bast3002 with bast3004, remove from bastion list
operations/puppet : productioninstall_server: remove bast3002 from DHCP, decom
operations/dns : masterdecom bast3002 ipv6 reverse
operations/dns : masterDecom bast3002
operations/puppet : productionDecom bast3002
operations/puppet : productionbast3002: decom phase 1
operations/dns : masteresams: move prometheus to bast3004
operations/puppet : productionadd bast3004 to esams prometheus nodes
operations/puppet : productionsmokeping: replace bast3002 with bast3004 as target

Event Timeline

Dzahn changed the task status from Open to Stalled.Wed, Oct 23, 11:17 PM
Dzahn created this task.
Dzahn added subscribers: mark, Aklapper.

Note this host used to be called "hooft" and there was still a relabeling ticket open at T198790 (merged in). So part of this should be to remove any "hooft" remnants too.

Also a note for when the time comes: there's Prometheus data on this host that will need to be migrated onto a VM on esams' ganeti cluster once that's online

Change 545911 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: replace bast3002 with bast3004

https://gerrit.wikimedia.org/r/545911

Dzahn claimed this task.Thu, Oct 24, 6:28 PM

@fgiunchedi ACK, so no data needs to be copied from bast3002 to bast3004, the new bastion? Instead it moves to a VM?

Change 545921 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] smokeping: replace bast3002 with bast3004 as target

https://gerrit.wikimedia.org/r/545921

Dzahn added a comment.Thu, Oct 24, 7:03 PM

@fgiunchedi Is there already a ticket for setting up those new prometheus VMs? I see it needs some related changes in puppet, so i'll block this ticket on that.

Change 545921 merged by Dzahn:
[operations/puppet@production] smokeping: replace bast3002 with bast3004 as target

https://gerrit.wikimedia.org/r/545921

Mentioned in SAL (#wikimedia-operations) [2019-10-24T23:46:41Z] <mutante> bast3002 - rsyncing /home, /srv/tfptboot and /srv/prometheus to /srv/bast3002/ on bast3004 (T236394 T236329)

Dzahn added a comment.EditedThu, Oct 24, 11:49 PM

Also a note for when the time comes: there's Prometheus data on this host that will need to be migrated onto a VM on esams' ganeti cluster once that's online

I rsynced /srv/prometheus/ from bast3002 to bast3004 /srv/bast3002/prometheus/ so that we can unblock the decom of bast3002 before being able to have those new prometheus VMs (per talking with Brandon)

The command is:

[bast3002:/srv] $ sudo rsync -avp /srv/prometheus/ rsync://bast3004.wikimedia.org/prometheus/

puppet setup to allow the rsync is:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/545935

Change 546001 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] switch esams prometheus node from bast3002 to bast3004

https://gerrit.wikimedia.org/r/546001

Change 546004 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: remove bast3002 from DHCP, decom

https://gerrit.wikimedia.org/r/546004

Change 546001 merged by Filippo Giunchedi:
[operations/puppet@production] add bast3004 to esams prometheus nodes

https://gerrit.wikimedia.org/r/546001

Mentioned in SAL (#wikimedia-operations) [2019-10-25T08:45:34Z] <godog> stop prometheus on bast300[24] and done last round of rsync data - T236329

Change 546120 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] esams: move prometheus to bast3004

https://gerrit.wikimedia.org/r/546120

Change 546120 merged by Filippo Giunchedi:
[operations/dns@master] esams: move prometheus to bast3004

https://gerrit.wikimedia.org/r/546120

All data sync'd to bast3004 and DNS flipped, bast3002 can continue decom from my POV, thanks @Dzahn !

Change 546140 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] bast3002: decom phase 1

https://gerrit.wikimedia.org/r/546140

Change 546140 merged by Filippo Giunchedi:
[operations/puppet@production] bast3002: decom phase 1

https://gerrit.wikimedia.org/r/546140

Change 546175 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Set bast3002 to spare

https://gerrit.wikimedia.org/r/546175

cookbooks.sre.hosts.decommission executed by filippo@cumin1001 for hosts: bast3002.wikimedia.org

  • bast3002.wikimedia.org (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

Change 546175 merged by Filippo Giunchedi:
[operations/puppet@production] Decom bast3002

https://gerrit.wikimedia.org/r/546175

fgiunchedi updated the task description. (Show Details)Fri, Oct 25, 1:39 PM

Change 546184 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] Decom bast3002

https://gerrit.wikimedia.org/r/546184

Change 546184 merged by Filippo Giunchedi:
[operations/dns@master] Decom bast3002

https://gerrit.wikimedia.org/r/546184

fgiunchedi reassigned this task from Dzahn to Papaul.Fri, Oct 25, 1:46 PM
fgiunchedi updated the task description. (Show Details)

I took over from @Dzahn in the interest of time, @Papaul host is ready for on site steps!

Change 546201 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] decom bast3002 ipv6 reverse

https://gerrit.wikimedia.org/r/546201

Change 546201 merged by BBlack:
[operations/dns@master] decom bast3002 ipv6 reverse

https://gerrit.wikimedia.org/r/546201

Dzahn changed the task status from Stalled to Open.Fri, Oct 25, 3:14 PM

Change 546004 abandoned by Dzahn:
install_server: remove bast3002 from DHCP, decom

Reason:
duplicated

https://gerrit.wikimedia.org/r/546004

Change 545911 abandoned by Dzahn:
site: replace bast3002 with bast3004, remove from bastion list

Reason:
duplicated

https://gerrit.wikimedia.org/r/545911

Papaul updated the task description. (Show Details)Sun, Oct 27, 6:58 PM

Change 546428 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for bast3002

https://gerrit.wikimedia.org/r/546428

Change 546428 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for bast3002

https://gerrit.wikimedia.org/r/546428

Papaul closed this task as Resolved.Mon, Oct 28, 9:32 AM
Papaul updated the task description. (Show Details)

Complete