Page MenuHomePhabricator
Create Task
Maniphest T280203

decom 44 eqiad appservers purchased on 2016-04-12/13 (mw1261 through mw1301)
Closed, ResolvedPublic
Actions

Description

lowest number mw server in production is mw1261

4 mw appservers in eqiad that have been purchased on 2016-04-12 (procured in T127735):

rack A5

  • mw1261.eqiad.wmnet - canary
  • mw1262.eqiad.wmnet - canary
  • mw1263.eqiad.wmnet - canary
  • mw1264.eqiad.wmnet - canary

40 mw appservers in eqiad that have been purchased on 2016-04-13 (procured in T127735):

rack A5

  • mw1265.eqiad.wmnet - canary
  • mw1266.eqiad.wmnet

rack A8

  • mw1267.eqiad.wmnet
  • mw1268.eqiad.wmnet

rack A7

  • mw1269.eqiad.wmnet - scap proxy !
  • mw1270.eqiad.wmnet
  • mw1271.eqiad.wmnet - mcrouter proxy !
  • mw1272.eqiad.wmnet
  • mw1273.eqiad.wmnet
  • mw1274.eqiad.wmnet
  • mw1275.eqiad.wmnet
  • mw1276.eqiad.wmnet - canary api
  • mw1277.eqiad.wmnet - canary api
  • mw1278.eqiad.wmnet - canary api
  • mw1279.eqiad.wmnet - canary api

rack A8

  • mw1280.eqiad.wmnet - already died in T251077
  • mw1281.eqiad.wmnet
  • mw1282.eqiad.wmnet
  • mw1283.eqiad.wmnet

rack B6

  • mw1284.eqiad.wmnet
  • mw1285.eqiad.wmnet - scap proxy !
  • mw1286.eqiad.wmnet
  • mw1287.eqiad.wmnet
  • mw1288.eqiad.wmnet - mcrouter proxy !
  • mw1289.eqiad.wmnet - API appserver
  • mw1290.eqiad.wmnet - API appserver
  • mw1291.eqiad.wmnet - was reimaged as thumbor
  • mw1292.eqiad.wmnet - was reimaged as thumbor
  • mw1293.eqiad.wmnet - jobrunner, canary
  • mw1294.eqiad.wmnet - jobrunner, canary
  • mw1295.eqiad.wmnet - jobrunner
  • mw1296.eqiad.wmnet - jobrunner
  • mw1297.eqiad.wmnet - API appserver
  • mw1298.eqiad.wmnet - jobrunner
  • mw1299.eqiad.wmnet - jobrunner
  • mw1300.eqiad.wmnet
  • mw1301.eqiad.wmnet

It adds up to 44 but 46 were in procurement ticket. (check the diff)

These should be replaced by new hardware (45/43 servers) procured in T271155 and coming up in T273915 and T279309.

They need to be decom'ed in parallel with setting up new servers in T279309.

Details

SubjectRepoBranchLines +/-
mw130[2-6]: Remove and decomissionoperations/puppetproduction+3 -13
scap: Switch mw1306 to mw1318 for scap proxy roleoperations/puppetproduction+1 -1
hiera/appservers: remove mcrouter proxy values for eqiadoperations/puppetproduction+0 -90
hieradata: cleanup old canary api serveroperations/puppetproduction+4 -899
site/conftool: remove mw1276 through mw1279operations/puppetproduction+0 -9
site/conftool: decom mw1271 and mw1288, formerly mcrouter proxiesoperations/puppetproduction+0 -12
site/conftool: decom jobrunners: mw1295,mw1296,mw1298,mw1299operations/puppetproduction+1 -5
site/conftool: convert 4 appservers to jobrunners in row D for balanceoperations/puppetproduction+13 -11
DHCP: remove mw1285 through mw1301operations/puppetproduction+0 -25
site/conftool/scap: decom mw1269, replace scap proxy with mw1420operations/puppetproduction+2 -3
site/conftool/DHCP: decom mw1285operations/puppetproduction+1 -7
site/conftool/DHCP: decom mw1293, mw1294, jobrunner canariesoperations/puppetproduction+1 -13
site/conftool/DHCP: decom mw1289, mw1290, mw1297operations/puppetproduction+1 -19
site/conftool/DHCP: decom mw1284,mw1286,mw1287operations/puppetproduction+1 -14
site/conftool: decom mw1270,mw1272,mw1273,mw1274,mw1275operations/puppetproduction+1 -6
DHCP/conftool: remove mw1300, mw1301operations/puppetproduction+0 -12
site/conftool: remove mw1281 through mw1283operations/puppetproduction+0 -8
logstash/tests: also replace IP of mw1281 with IP of mw1391operations/puppetproduction+1 -1
logstash/tests: replace mw1281 with mw1391operations/puppetproduction+2 -2
DHCP: remove mw1269 through mw1284operations/puppetproduction+0 -75
site/install/conftool: decom mw1267, mw1268operations/puppetproduction+0 -17
remove hieradata/hosts files for former eqiad canariesoperations/puppetproduction+0 -1 K
site/conftool/DHCP: remove old eqiad appservers in A5, mostly canariesoperations/puppetproduction+0 -46
trafficserver: remove comment about a server that won't exist anymoreoperations/puppetproduction+1 -2
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jul 14 2021, 2:47 PM

Change 704284 merged by Dzahn:

[operations/puppet@production] site/conftool: remove mw1281 through mw1283

https://gerrit.wikimedia.org/r/704284

ops-monitoring-bot added a comment.Jul 15 2021, 1:34 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1300.eqiad.wmnet

  • mw1300.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Dzahn raised the priority of this task from Medium to High.Jul 15 2021, 1:34 PM
Dzahn updated the task description. (Show Details)
Jelto updated the task description. (Show Details)Jul 15 2021, 1:58 PM
Jelto updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)Jul 15 2021, 1:59 PM
ops-monitoring-bot added a comment.Jul 15 2021, 2:10 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1301.eqiad.wmnet

  • mw1301.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jul 15 2021, 2:18 PM

Change 704289 merged by Dzahn:

[operations/puppet@production] DHCP/conftool: remove mw1300, mw1301

https://gerrit.wikimedia.org/r/704289

gerritbot added a comment.Jul 16 2021, 1:13 PM

Change 704966 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: decom mw1270,mw1272,mw1273,mw1274,mw1275

https://gerrit.wikimedia.org/r/704966

ops-monitoring-bot added a comment.Jul 19 2021, 1:21 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1270.eqiad.wmnet

  • mw1270.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Jul 19 2021, 1:42 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1272.eqiad.wmnet

  • mw1272.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Jul 19 2021, 1:57 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[1273-1275].eqiad.wmnet

  • mw1273.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • mw1274.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • mw1275.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jul 19 2021, 2:06 PM

Change 704966 merged by Dzahn:

[operations/puppet@production] site/conftool: decom mw1270,mw1272,mw1273,mw1274,mw1275

https://gerrit.wikimedia.org/r/704966

Dzahn updated the task description. (Show Details)Jul 19 2021, 2:14 PM
ops-monitoring-bot added a comment.Jul 19 2021, 4:46 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1284.eqiad.wmnet

  • mw1284.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Jul 19 2021, 5:13 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[1286-1287].eqiad.wmnet

  • mw1286.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • mw1287.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jul 19 2021, 6:19 PM

Change 705439 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool/DHCP: decom mw1284,mw1286,mw1287

https://gerrit.wikimedia.org/r/705439

gerritbot added a comment.Jul 19 2021, 6:23 PM

Change 705439 merged by Dzahn:

[operations/puppet@production] site/conftool/DHCP: decom mw1284,mw1286,mw1287

https://gerrit.wikimedia.org/r/705439

Dzahn updated the task description. (Show Details)Jul 19 2021, 6:43 PM
Dzahn updated the task description. (Show Details)Jul 20 2021, 3:06 PM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Jul 20 2021, 3:11 PM

Change 705700 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool/DHCP: decom mw1289, mw1290, mw1297

https://gerrit.wikimedia.org/r/705700

ops-monitoring-bot added a comment.Jul 20 2021, 4:10 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1289.eqiad.wmnet

  • mw1289.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Jul 20 2021, 4:21 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1290.eqiad.wmnet

  • mw1290.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Dzahn updated the task description. (Show Details)Jul 20 2021, 4:27 PM
Dzahn updated the task description. (Show Details)Jul 20 2021, 4:29 PM
ops-monitoring-bot added a comment.Jul 20 2021, 4:37 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1297.eqiad.wmnet

  • mw1297.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jul 20 2021, 4:38 PM

Change 705700 merged by Dzahn:

[operations/puppet@production] site/conftool/DHCP: decom mw1289, mw1290, mw1297

https://gerrit.wikimedia.org/r/705700

Jelto mentioned this in T233196: Migrate thumbor to Kubernetes .Jul 21 2021, 9:01 AM
gerritbot added a comment.Jul 21 2021, 1:16 PM

Change 705875 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool/DHCP: decom mw1293, mw1294, jobrunner canaries

https://gerrit.wikimedia.org/r/705875

ops-monitoring-bot added a comment.Jul 21 2021, 1:49 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1293.eqiad.wmnet

  • mw1293.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Jul 21 2021, 2:07 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1294.eqiad.wmnet

  • mw1294.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jul 21 2021, 2:13 PM

Change 705875 merged by Dzahn:

[operations/puppet@production] site/conftool/DHCP: decom mw1293, mw1294, jobrunner canaries

https://gerrit.wikimedia.org/r/705875

Dzahn updated the task description. (Show Details)Jul 21 2021, 2:27 PM
Dzahn updated the task description. (Show Details)Jul 27 2021, 9:03 AM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Jul 27 2021, 9:11 AM

Change 708253 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool/DHCP: decom mw1285

https://gerrit.wikimedia.org/r/708253

Dzahn updated the task description. (Show Details)Jul 27 2021, 9:16 AM
gerritbot added a comment.Jul 27 2021, 9:22 AM

Change 708254 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool/scap: decom mw1269, replace scap proxy with mw1420

https://gerrit.wikimedia.org/r/708254

gerritbot added a comment.Jul 27 2021, 9:34 AM

Change 708257 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] hiera/appservers: remove mcrouter proxy values for eqiad

https://gerrit.wikimedia.org/r/708257

ops-monitoring-bot added a comment.Jul 27 2021, 9:41 AM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1285.eqiad.wmnet

  • mw1285.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jul 27 2021, 9:50 AM

Change 708253 merged by Dzahn:

[operations/puppet@production] site/conftool/DHCP: decom mw1285

https://gerrit.wikimedia.org/r/708253

gerritbot added a comment.Jul 27 2021, 10:09 AM

Change 708254 merged by Dzahn:

[operations/puppet@production] site/conftool/scap: decom mw1269, replace scap proxy with mw1420

https://gerrit.wikimedia.org/r/708254

ops-monitoring-bot added a comment.Jul 27 2021, 10:39 AM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1269.eqiad.wmnet

  • mw1269.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Dzahn updated the task description. (Show Details)Jul 27 2021, 11:01 AM
Dzahn updated the task description. (Show Details)Jul 28 2021, 11:08 AM
Dzahn added a comment.Jul 28 2021, 11:12 AM

@wiki_willy You guys can remove all old mw appservers from eqiad rack A5 and rack A8 already, they are decom'ed:

rack A5 mw1261 through mw1266

rack A8 mw1267, mw1268, mw1280 through mw1283

26 servers are in status "decommissioning" now in the "mw12" range:

https://netbox.wikimedia.org/dcim/devices/?q=mw12&status=decommissioning&mac_address=&has_primary_ip=&local_context_data=&virtual_chassis_member=&console_ports=&console_server_ports=&power_ports=&power_outlets=&interfaces=&pass_through_ports=&cf_purchase_date=

gerritbot added a comment.Jul 28 2021, 12:13 PM

Change 679958 merged by Dzahn:

[operations/puppet@production] DHCP: remove mw1285 through mw1301

https://gerrit.wikimedia.org/r/679958

wiki_willy added a comment.Jul 28 2021, 5:45 PM

Thanks @Dzahn!

>>! In T280203#7242737, @Dzahn wrote:

@wiki_willy You guys can remove all old mw appservers from eqiad rack A5 and rack A8 already, they are decom'ed:

rack A5 mw1261 through mw1266

rack A8 mw1267, mw1268, mw1280 through mw1283

26 servers are in status "decommissioning" now in the "mw12" range:

https://netbox.wikimedia.org/dcim/devices/?q=mw12&status=decommissioning&mac_address=&has_primary_ip=&local_context_data=&virtual_chassis_member=&console_ports=&console_server_ports=&power_ports=&power_outlets=&interfaces=&pass_through_ports=&cf_purchase_date=

RKemper mentioned this in T281989: Q4:(Need By: TBD) rack/setup/install elastic10[68-83].eqiad.wmnet.Jul 28 2021, 9:12 PM
gerritbot added a comment.Jul 30 2021, 1:36 PM

Change 709041 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: convert 4 appservers to jobrunners in row D for balance

https://gerrit.wikimedia.org/r/709041

gerritbot added a comment.Jul 30 2021, 1:51 PM

Change 709041 merged by Dzahn:

[operations/puppet@production] site/conftool: convert 4 appservers to jobrunners in row D for balance

https://gerrit.wikimedia.org/r/709041

gerritbot added a comment.Jul 30 2021, 3:06 PM

Change 709068 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: decom jobrunners: mw1295,mw1296,mw1298,mw1299

https://gerrit.wikimedia.org/r/709068

ops-monitoring-bot added a comment.Jul 30 2021, 3:17 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[1295-1296].eqiad.wmnet

  • mw1295.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • mw1296.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Jul 30 2021, 3:33 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[1298-1299].eqiad.wmnet

  • mw1298.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • mw1299.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jul 30 2021, 3:44 PM

Change 709068 merged by Dzahn:

[operations/puppet@production] site/conftool: decom jobrunners: mw1295,mw1296,mw1298,mw1299

https://gerrit.wikimedia.org/r/709068

Dzahn updated the task description. (Show Details)Jul 30 2021, 3:45 PM
gerritbot added a comment.Aug 2 2021, 11:24 AM

Change 709411 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: decom mw1271 and mw1288, formerly mcrouter proxies

https://gerrit.wikimedia.org/r/709411

ops-monitoring-bot added a comment.Aug 2 2021, 11:42 AM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1271.eqiad.wmnet

  • mw1271.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Aug 2 2021, 11:54 AM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw1288.eqiad.wmnet

  • mw1288.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Aug 2 2021, 11:58 AM

Change 709411 merged by Dzahn:

[operations/puppet@production] site/conftool: decom mw1271 and mw1288, formerly mcrouter proxies

https://gerrit.wikimedia.org/r/709411

Dzahn claimed this task.Aug 2 2021, 11:59 AM
Dzahn updated the task description. (Show Details)
Dzahn changed the task status from Open to Stalled.Aug 2 2021, 12:04 PM

everything is done except 4 canary API servers and decom'ing these is blocked by T273915

wiki_willy added a subtask: T276743: ps1-a7-eqiad power over threshold alerts.Aug 2 2021, 11:10 PM
• Cmjohnson added a project: ops-eqiad.Aug 4 2021, 4:44 PM
• Cmjohnson added a comment.Aug 6 2021, 6:54 PM

decom script executed and servers removed from racks for
mw1261-1266 rack A5
mw1269-1275 rack A7

• Cmjohnson closed subtask T276743: ps1-a7-eqiad power over threshold alerts as Resolved.Aug 6 2021, 7:30 PM
• Cmjohnson added a comment.Aug 11 2021, 7:45 PM

mw1267-mw1268 decom'd and removed

Dzahn changed the task status from Stalled to Open.Aug 17 2021, 7:53 AM
gerritbot added a comment.Aug 17 2021, 8:14 AM

Change 704287 merged by Dzahn:

[operations/puppet@production] site/conftool: remove mw1276 through mw1279

https://gerrit.wikimedia.org/r/704287

Stashbot added a comment.Aug 17 2021, 8:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-17T08:15:23Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mw[1276-1279].eqiad.wmnet with reason: decom old appservers in eqiad T280203

Stashbot added a comment.Aug 17 2021, 8:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-17T08:15:32Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw[1276-1279].eqiad.wmnet with reason: decom old appservers in eqiad T280203

ops-monitoring-bot added a comment.Aug 17 2021, 8:37 AM

cookbooks.sre.hosts.decommission executed by jelto@cumin1001 for hosts: mw[1276-1279].eqiad.wmnet

  • mw1276.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • mw1277.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • mw1278.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • mw1279.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Jelto updated the task description. (Show Details)Aug 17 2021, 8:39 AM
Dzahn reassigned this task from Dzahn to Jelto.EditedAug 17 2021, 9:17 AM

Jelto, over to you, since you are removing the last 4 servers :) Just cause I know you still wanted to check something in the repo one last time. So you can decide if it's closed.

gerritbot added a comment.Aug 18 2021, 8:30 AM

Change 713607 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] hieradata::hosts::mw1 cleanup old canary api server hieradata

https://gerrit.wikimedia.org/r/713607

gerritbot added a comment.Aug 18 2021, 2:11 PM

Change 713607 merged by Jelto:

[operations/puppet@production] hieradata: cleanup old canary api server

https://gerrit.wikimedia.org/r/713607

Jelto added a comment.Aug 18 2021, 2:41 PM

From my side everything is done. Thanks everyone. I'm going to close this ticket. Feel free to open it in case I missed anything.

I would like to re-open https://phabricator.wikimedia.org/T221132 to have a short discussion about the refresh of thumbor hosts. We just skipped the two hosts during our refresh. I don't want that we forget about the two hosts completely.

Jelto closed this task as Resolved.Aug 18 2021, 2:42 PM
Jelto mentioned this in Unknown Object (Task).
Dzahn awarded a token.Aug 18 2021, 2:47 PM
• Cmjohnson reopened this task as Open.Aug 18 2021, 4:00 PM
• Cmjohnson claimed this task.

re-opened and assigned to me to use this same task as the final decommissioning inside the data center

• Cmjohnson closed this task as Resolved.Aug 18 2021, 5:04 PM

all are decommissioned and removed from the rack

gerritbot added a comment.Sep 27 2021, 8:19 PM

Change 708257 abandoned by Dzahn:

[operations/puppet@production] hiera/appservers: remove mcrouter proxy values for eqiad

Reason:

obsolete

https://gerrit.wikimedia.org/r/708257

gerritbot added a comment.Mar 3 2022, 1:33 PM

Change 767787 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] scap: Switch mw1306 to mw1318 for scap proxy role

https://gerrit.wikimedia.org/r/767787

gerritbot added a comment.Mar 3 2022, 1:33 PM

Change 767788 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] mw130[2-6]: Remove and decomission

https://gerrit.wikimedia.org/r/767788

wiki_willy mentioned this in T302034: eqiad: Unrack wmf3570, wmf4579, conf1003, mw1301.Mar 3 2022, 5:56 PM
gerritbot added a comment.Mar 4 2022, 8:45 AM

Change 767787 merged by Alexandros Kosiaris:

[operations/puppet@production] scap: Switch mw1306 to mw1318 for scap proxy role

https://gerrit.wikimedia.org/r/767787

gerritbot added a comment.Mar 4 2022, 1:12 PM

Change 767788 merged by Alexandros Kosiaris:

[operations/puppet@production] mw130[2-6]: Remove and decomission

https://gerrit.wikimedia.org/r/767788

ayounsi reopened subtask T276743: ps1-a7-eqiad power over threshold alerts as Open.Aug 29 2022, 12:39 PM
Jclark-ctr closed subtask T276743: ps1-a7-eqiad power over threshold alerts as Resolved.Sep 23 2022, 4:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits