We added new servers on codfw T247021 but we didn't decomm the hosts scheduled to retire. With the upcoming DC switchover we need to speed this up as keeping those hosts in the cluster will just increase the overall latency
This ticket is about all Dell PowerEdge R420 servers that have been procured in RT ticket #9011 in 2015 aka "type A hardware".
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | Papaul | T241852 (Need by: TBD) rack/setup/install 86 new codfw mw systems | |||
| Resolved | Dzahn | T247021 move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet) | |||
| Resolved | RLazarus | T243314 FY2020-2021 Q1 DC switchover and switchback | |||
| Resolved | RLazarus | T243316 FY2020-2021 Q1 eqiad -> codfw switchover | |||
| Resolved | Dzahn | T260654 Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420) | |||
| Resolved | Papaul | T261524 decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part |
Change 621783 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] decom mw2135 through mw2214
Change 621786 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] decom mw2135 through mw2214
https://docs.google.com/spreadsheets/d/1rtg4DMx4glZA6T_XVLzt_OlFHQx53Eb_U8criLzCQs4/edit?usp=sharing shows in the TOTALs sheet how this affects the balance of servers between eqiad and codfw. It is still around 50% of servers in each DC.
High.. _if_ we want it to happen before the switch. We could of course also just set them to "inactive" / weight 0 and do the rest later but have the same effect for traffic.
@Dzahn if you're going ahead with this please give me a heads up as I have a patch to merge for the decom cookbook and would like to see it works fine in real life
@Volans Sure, I am currently just waiting for the ok from other subscribers here (https://gerrit.wikimedia.org/r/c/operations/puppet/+/621783)
Mentioned in SAL (#wikimedia-operations) [2020-08-27T16:48:09Z] <mutante> depooling mw2187 - mw2199 - old codfw appservers of type A to be decom'ed, previously weight 10 (T260654)
Change 621783 merged by Dzahn:
[operations/puppet@production] decom mw2187-mw2199, mw2135-mw2147, mw2200-mw2214
Change 622879 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP: remove mw2187-mw2199, mw2135-mw2147, mw2200-mw2214
Change 622879 merged by Dzahn:
[operations/puppet@production] DHCP: remove mw2187-mw2199, mw2135-mw2147, mw2200-mw2214
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2187.codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2189.codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[2190-2194].codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2195.codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[2197-2199].codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[2135-2139].codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[2140-2144].codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[2145-2147].codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[2200-2204].codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[2205-2209].codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw[2210-2212,2214].codfw.wmnet
Change 622895 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: remove proxy for codfw C4, mw2188
Change 622895 merged by Dzahn:
[operations/puppet@production] scap: remove proxy for codfw C4, mw2188
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2188.codfw.wmnet
Change 622900 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki: replace mw2196 with mw2336 as mcrouter proxy
Change 622902 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: remove mw2187-mw2195, mw2197-mw2199, mw2135-mw2147, mw2200-mw2214
Change 622902 merged by Dzahn:
[operations/puppet@production] site: remove mw2187-mw2195, mw2197-mw2199, mw2135-mw2147, mw2200-mw2214
Change 622906 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: remove mw2189-mw2195,mw2197-mw2199
Change 622906 merged by Dzahn:
[operations/puppet@production] site: remove mw2189-mw2195,mw2197-mw2199
Change 622900 merged by Dzahn:
[operations/puppet@production] mediawiki: replace mw2196 with mw2336 as mcrouter proxy
Change 623022 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: remove mw2196.codfw.wmnet
cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: mw2196.codfw.wmnet
Change 623022 merged by Dzahn:
[operations/puppet@production] site: remove mw2196.codfw.wmnet
Change 623031 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove mw2135-mw2147 and mw2187-mw2214
Change 623031 merged by Dzahn:
[operations/dns@master] remove mw2135-mw2147 and mw2187-mw2214
All steps that need to be done by us as the service owner are done.
40 hosts removed from repos, shutdown and wiped bootloader by decom cookbook, production IPs removed.
I made a separate ticket to hand-over to dcops because that needs 40 x the decom template, which means 680 checkboxes :)
to be continued on T261524
Change 621786 abandoned by Dzahn:
[operations/dns@master] decom mw2135 through mw2214
Reason:
duplicate of https://gerrit.wikimedia.org/r/c/operations/dns/ /623031