Page MenuHomePhabricator

decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242)
Closed, ResolvedPublic

Description

From context on T274171 it seems we need to decom all codfw appservers purchased on 2016-05-17.

Is this right?

That is currently 28 boxes reported by netbox. It does not match exactly the procurement task T134272 though.

  • mw2215
  • mw2216
  • mw2217
  • mw2218
  • mw2219
  • mw2220
  • mw2221
  • mw2222
  • mw2223
  • mw2224
  • mw2225
  • mw2226
  • mw2227
  • mw2228
  • mw2229
  • mw2230
  • mw2231
  • mw2232
  • mw2233
  • mw2234
  • mw2235
  • mw2236
  • mw2237
  • mw2238
  • mw2239
  • mw2240
  • mw2241
  • mw2242

I do not want to repeat the same template 28 times on this ticket to end up with hundreds of checkboxes, but we need to follow the server lifecycle page anyways.

Event Timeline

Dzahn renamed this task from decom 28 codfw appservers purchased on 2016-05-17 to decom 28 codfw appservers purchased on 2016-05-17 (mw2215 through mw2242).Mar 11 2021, 12:47 AM
Dzahn updated the task description. (Show Details)

As @Papaul points out mw2241 to mw2250 also count as old ( a separate purchase a few days later in 2016) but mw2251 does not (2016 but December). So the cut-off is in 2016 after May but before December?

Change 670631 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: replace mw2215,mw2216 with mw2377,mw2378 as codfw API canaries

https://gerrit.wikimedia.org/r/670631

Mentioned in SAL (#wikimedia-operations) [2021-03-11T20:58:10Z] <mutante> deactivating codfw API canaries on old hardware (T277119)

Change 670944 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: switch proxy for codfw A3 from mw2215 to mw2300

https://gerrit.wikimedia.org/r/670944

Change 670949 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mcrouter: replace proxy for codfw A3, mw2235->mw2300

https://gerrit.wikimedia.org/r/670949

Change 670953 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP: remove mw2215 through mw2242

https://gerrit.wikimedia.org/r/670953

Change 670953 merged by Dzahn:
[operations/puppet@production] DHCP: remove mw2215 through mw2242

https://gerrit.wikimedia.org/r/670953

Change 670957 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: remove mw2224 through mw2229

https://gerrit.wikimedia.org/r/670957

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2216.codfw.wmnet

  • mw2216.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

Mentioned in SAL (#wikimedia-operations) [2021-03-11T22:55:57Z] <mutante> depooled mw2224 through mw2228 but not removing from DSH groups yet (T277119)

Change 670944 merged by Dzahn:
[operations/puppet@production] scap: switch proxy for codfw A3 from mw2215 to mw2300

https://gerrit.wikimedia.org/r/670944

Change 670992 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool: turn mw2274 and mw2276 into canaries

https://gerrit.wikimedia.org/r/670992

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2215.codfw.wmnet

  • mw2215.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

Change 670631 merged by Dzahn:
[operations/puppet@production] site: decom mw2215,mw2216, codfw API canaries

https://gerrit.wikimedia.org/r/670631

Mentioned in SAL (#wikimedia-operations) [2021-03-12T18:03:50Z] <mutante> depooling mw2244,mw2245 (API on old hardware), mw2229,mw2230 (app on old hardware) - T277119

Change 671212 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool: decom mw2217, mw2218, mw2219

https://gerrit.wikimedia.org/r/671212

Change 670992 merged by Dzahn:
[operations/puppet@production] site/conftool: turn mw2374 and mw2376 into API canaries

https://gerrit.wikimedia.org/r/670992

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2217.codfw.wmnet

  • mw2217.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2218.codfw.wmnet

  • mw2218.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2219.codfw.wmnet

  • mw2219.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

Change 671212 merged by Dzahn:
[operations/puppet@production] site/conftool: decom mw2217, mw2218, mw2219

https://gerrit.wikimedia.org/r/671212

Dzahn triaged this task as High priority.Mar 12 2021, 9:17 PM
Dzahn updated the task description. (Show Details)
Dzahn set Due Date to Mar 22 2021, 7:00 AM.
Dzahn moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.

Change 671259 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool: decom mw2220 through mw2223

https://gerrit.wikimedia.org/r/671259

Change 671260 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool: decom mw2239 through mw2242

https://gerrit.wikimedia.org/r/671260

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2224.codfw.wmnet

  • mw2224.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run Homer on asw-a-codfw.mgmt.codfw.wmnet: Command '['/usr/local/bin/homer', 'asw-a-codfw.mgmt.codfw.wmnet', 'commit', 'Host decommission - dzahn@cumin1001 - T277119']' returned non-zero exit status 1.

ERROR: some step on some host failed, check the bolded items above

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2220.codfw.wmnet

  • mw2220.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2221.codfw.wmnet

  • mw2221.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2222.codfw.wmnet

  • mw2222.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

Change 671259 merged by Dzahn:
[operations/puppet@production] site/conftool: decom mw2220 through mw2223

https://gerrit.wikimedia.org/r/671259

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2223.codfw.wmnet

  • mw2223.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2225.codfw.wmnet

  • mw2225.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

Change 670949 merged by Dzahn:
[operations/puppet@production] mcrouter: replace proxy for codfw A3, mw2235->mw2299

https://gerrit.wikimedia.org/r/670949

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2226.codfw.wmnet

  • mw2226.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run Homer on asw-a-codfw.mgmt.codfw.wmnet: Command '['/usr/local/bin/homer', 'asw-a-codfw.mgmt.codfw.wmnet', 'commit', 'Host decommission - dzahn@cumin1001 - T277119']' returned non-zero exit status 1.

ERROR: some step on some host failed, check the bolded items above

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2227.codfw.wmnet

  • mw2227.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run Homer on asw-a-codfw.mgmt.codfw.wmnet: Command '['/usr/local/bin/homer', 'asw-a-codfw.mgmt.codfw.wmnet', 'commit', 'Host decommission - dzahn@cumin1001 - T277119']' returned non-zero exit status 1.

ERROR: some step on some host failed, check the bolded items above

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2228.codfw.wmnet

  • mw2228.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2229.codfw.wmnet

  • mw2229.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

Change 670957 merged by Dzahn:
[operations/puppet@production] site/conftool: decom mw2224 through mw2229

https://gerrit.wikimedia.org/r/670957

Change 672834 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool: remove mw2230 through mw2242

https://gerrit.wikimedia.org/r/672834

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2230.codfw.wmnet

  • mw2230.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2231.codfw.wmnet

  • mw2231.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2232.codfw.wmnet

  • mw2232.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2233.codfw.wmnet

  • mw2233.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2234.codfw.wmnet

  • mw2234.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2235.codfw.wmnet

  • mw2235.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2236.codfw.wmnet

  • mw2236.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

Change 671260 abandoned by Dzahn:
[operations/puppet@production] site/conftool: decom mw2239 through mw2242

Reason:
duplicate

https://gerrit.wikimedia.org/r/671260

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2237.codfw.wmnet

  • mw2237.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2238.codfw.wmnet

  • mw2238.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

@Papaul rack A3 is ready. All old hosts in it have been shut down. The remaining mw servers in A3 (mw2291 and up) are modern hardware from 2019 that should stay.

But all the ones up to mw2238 are old and ready to be removed. See:

https://netbox.wikimedia.org/dcim/devices/?q=mw2&rack_id=45&mac_address=&has_primary_ip=&local_context_data=&virtual_chassis_member=&console_ports=&console_server_ports=&power_ports=&power_outlets=&interfaces=&pass_through_ports=&cf_purchase_date=

Change 672834 merged by Dzahn:
[operations/puppet@production] site/conftool: remove mw2230 through mw2238, rack A3

https://gerrit.wikimedia.org/r/672834

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2239.codfw.wmnet

  • mw2239.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2240.codfw.wmnet

  • mw2240.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2241.codfw.wmnet

  • mw2241.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2242.codfw.wmnet

  • mw2242.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

Change 673334 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftool-data: decom mw2239 through mw2242, rack A4

https://gerrit.wikimedia.org/r/673334

Change 673334 merged by Dzahn:
[operations/puppet@production] site/conftool-data: decom mw2239 through mw2242, rack A4

https://gerrit.wikimedia.org/r/673334

Dzahn updated the task description. (Show Details)
Papaul claimed this task.

Reopen this task since i have to offline all the servers in Netbox and remove the disks

Sure, I wasn't sure if you prefer your own ticket or this. Either is fine with me.

asset tag mgmt removed and all the disk removed also for server