Page MenuHomePhabricator
Create Task
Maniphest T247021

move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet)
Closed, ResolvedPublic
Actions

Description

After all 86 new servers (mw2[291-2377].codfw.wmnet) have been racked in T241852 by dcops, this is where serviceops takes over and does the steps to move them into production until they are actually pooled and serve traffic.

We want a separate ticket because different people are doing it and otherwise tickets stay open on the dcops workboard even though their part is done.

15 servers are blocked by T247018

mw2291 through mw2324 are npooled and status active in netbox (34 servers) https://gerrit.wikimedia.org/r/q/topic:%22appservers-codfw%22+(status:open%20OR%20status:merged)

mw2325 through mw2334 are pooled and status active in netbox (10 servers) https://gerrit.wikimedia.org/r/c/operations/puppet/+/577408

mw2335 through mw2349 are not pooled, not in site.pp and status planned in netbox (15 servers) (blocked by T247018)

mw2350 through mw2376 are pooled, in site.pp and and status active in netbox ((27 servers) (https://gerrit.wikimedia.org/r/c/operations/puppet/+/577409)

total: 86 servers

Details

SubjectRepoBranchLines +/-
conftool: add mw2335 - mw2339operations/puppetproduction+5 -0
add fake mcrouter certs for mw2335 - mw2339labs/privatemaster+0 -0
site: add new appservers mw2335 through mw2339operations/puppetproduction+5 -1
add management and production IPs for mw2335-mw2339operations/dnsmaster+31 -4
site: fix duplicate regex and row for mw2366-mw2376operations/puppetproduction+14 -7
add mw2350-2376 as API and appservers, codfw rack C6operations/puppetproduction+35 -3
add mw2325-mw2334 as API and appservers, codfw rack B6operations/puppetproduction+18 -3
site: add mw2301 through mw2309 as api and appserversoperations/puppetproduction+17 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Mar 5 2020, 7:29 PM

Only 71 of 86 servers can be done until T247018 is resolved first because there is not enough rack space for them.

Dzahn mentioned this in T241852: (Need by: TBD) rack/setup/install 86 new codfw mw systems.Mar 5 2020, 7:33 PM
gerritbot added a comment.Mar 6 2020, 12:11 AM

Change 577388 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add mw2301 through mw2309 as api and appservers

https://gerrit.wikimedia.org/r/577388

gerritbot added a project: Patch-For-Review.Mar 6 2020, 12:11 AM
ops-monitoring-bot added a comment.Mar 6 2020, 12:19 AM

Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 9 host(s) and their services with reason: new_install

mw[2301-2309].codfw.wmnet
gerritbot added a comment.Mar 6 2020, 12:20 AM

Change 577388 merged by Dzahn:
[operations/puppet@production] site: add mw2301 through mw2309 as api and appservers

https://gerrit.wikimedia.org/r/577388

ops-monitoring-bot added a comment.Mar 6 2020, 1:01 AM

Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 9 host(s) and their services with reason: new_install

mw[2301-2309].codfw.wmnet
Maintenance_bot removed a project: Patch-For-Review.Mar 6 2020, 1:10 AM
Stashbot added a comment.Mar 6 2020, 1:38 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-06T01:38:12Z] <mutante> added 9 more appservers to codfw pool split between appserver and API appservers, weight 15 (like all in codfw) T247021

Dzahn added a comment.Mar 6 2020, 1:41 AM

mw2301 thru mw2309 pooled and set to active in netbox

Dzahn renamed this task from move all 86 new codfw appservers into production to move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet).Mar 6 2020, 1:42 AM
Dzahn updated the task description. (Show Details)
Dzahn added a comment.EditedMar 6 2020, 1:51 AM

mw2291 through mw2324 are now pooled and status active in netbox (34 servers)

mw2325 through mw2334 are not pooled but in site.pp and status staged in netbox (10 servers)

mw2335 through mw2349 are not pooled, not in site.pp and status planned in netbox (15 servers) (blocked by T247018)

mw2350 through mw2376 are not pooled, in site.pp and and status staged in netbox ((27 servers)

total: 86 servers

gerritbot added a comment.Mar 6 2020, 2:00 AM

Change 577408 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add mw2325-mw2334 as API and appservers, codfw rack B6

https://gerrit.wikimedia.org/r/577408

gerritbot added a project: Patch-For-Review.Mar 6 2020, 2:00 AM
Dzahn updated the task description. (Show Details)Mar 6 2020, 2:03 AM
gerritbot added a comment.Mar 6 2020, 2:11 AM

Change 577409 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add mw2350-2376 as API and appservers, codfw rack C6

https://gerrit.wikimedia.org/r/577409

Dzahn claimed this task.Mar 6 2020, 2:17 AM
Dzahn triaged this task as High priority.
Dzahn updated the task description. (Show Details)Mar 6 2020, 2:22 AM
ops-monitoring-bot added a comment.Mar 6 2020, 4:40 PM

Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 5 host(s) and their services with reason: new_install

mw[2325-2329].codfw.wmnet
ops-monitoring-bot added a comment.Mar 6 2020, 4:40 PM

Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 4 host(s) and their services with reason: new_install

mw[2331-2334].codfw.wmnet
gerritbot added a comment.Mar 6 2020, 4:42 PM

Change 577408 merged by Dzahn:
[operations/puppet@production] add mw2325-mw2334 as API and appservers, codfw rack B6

https://gerrit.wikimedia.org/r/577408

ops-monitoring-bot added a comment.Mar 6 2020, 5:26 PM

Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 4 host(s) and their services with reason: new_install

mw[2331-2334].codfw.wmnet
ops-monitoring-bot added a comment.Mar 6 2020, 5:28 PM

Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 5 host(s) and their services with reason: new_install

mw[2325-2329].codfw.wmnet
Dzahn added a comment.Mar 6 2020, 6:09 PM
{"mw2325.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=appserver,service=apache2"}
{"mw2325.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=appserver,service=nginx"}
{"mw2326.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=api_appserver,service=apache2"}
{"mw2326.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=api_appserver,service=nginx"}
{"mw2327.codfw.wmnet": {"pooled": "yes", "weight": 15}, "tags": "dc=codfw,cluster=appserver,service=apache2"}
{"mw2327.codfw.wmnet": {"pooled": "yes", "weight": 15}, "tags": "dc=codfw,cluster=appserver,service=nginx"}
{"mw2328.codfw.wmnet": {"pooled": "yes", "weight": 15}, "tags": "dc=codfw,cluster=api_appserver,service=nginx"}
{"mw2328.codfw.wmnet": {"pooled": "yes", "weight": 15}, "tags": "dc=codfw,cluster=api_appserver,service=apache2"}
{"mw2329.codfw.wmnet": {"pooled": "yes", "weight": 15}, "tags": "dc=codfw,cluster=appserver,service=apache2"}
{"mw2329.codfw.wmnet": {"pooled": "yes", "weight": 15}, "tags": "dc=codfw,cluster=appserver,service=nginx"}
{"mw2330.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=api_appserver,service=apache2"}
{"mw2330.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=api_appserver,service=nginx"}
{"mw2331.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=appserver,service=apache2"}
{"mw2331.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=appserver,service=nginx"}
{"mw2332.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=api_appserver,service=nginx"}
{"mw2332.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=api_appserver,service=apache2"}
{"mw2333.codfw.wmnet": {"pooled": "yes", "weight": 15}, "tags": "dc=codfw,cluster=appserver,service=apache2"}
{"mw2333.codfw.wmnet": {"pooled": "yes", "weight": 15}, "tags": "dc=codfw,cluster=appserver,service=nginx"}
{"mw2334.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=api_appserver,service=apache2"}
{"mw2334.codfw.wmnet": {"weight": 15, "pooled": "yes"}, "tags": "dc=codfw,cluster=api_appserver,service=nginx"}
Dzahn added a comment.Mar 6 2020, 6:11 PM

mw2325 through mw2334 set to Active in Netbox

10 servers pooled at 18:05 UTC, March 6th.

Dzahn updated the task description. (Show Details)Mar 6 2020, 6:19 PM
ops-monitoring-bot added a comment.Mar 10 2020, 8:29 PM

Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 27 host(s) and their services with reason: new_install

mw[2350-2376].codfw.wmnet
gerritbot added a comment.Mar 10 2020, 8:33 PM

Change 577409 merged by Dzahn:
[operations/puppet@production] add mw2350-2376 as API and appservers, codfw rack C6

https://gerrit.wikimedia.org/r/577409

Maintenance_bot removed a project: Patch-For-Review.Mar 10 2020, 9:10 PM
ops-monitoring-bot added a comment.Mar 10 2020, 9:29 PM

Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 27 host(s) and their services with reason: new_install

mw[2350-2376].codfw.wmnet
gerritbot added a comment.Mar 10 2020, 10:31 PM

Change 578630 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: remove duplicate regex for mw2366-mw2376

https://gerrit.wikimedia.org/r/578630

gerritbot added a project: Patch-For-Review.Mar 10 2020, 10:31 PM
gerritbot added a comment.Mar 10 2020, 11:08 PM

Change 578630 merged by Dzahn:
[operations/puppet@production] site: fix duplicate regex and row for mw2366-mw2376

https://gerrit.wikimedia.org/r/578630

Maintenance_bot removed a project: Patch-For-Review.Mar 10 2020, 11:11 PM
ops-monitoring-bot added a comment.Mar 10 2020, 11:11 PM

Icinga downtime for 2:00:00 set by dzahn@cumin1001 on 6 host(s) and their services with reason: new_install

mw[2366,2368,2370,2372,2374,2376].codfw.wmnet
Dzahn added a comment.Mar 11 2020, 12:18 AM

mw2350 through mw2376 are all pooled in production and set to "Active" in netbox now.

Dzahn updated the task description. (Show Details)Mar 11 2020, 12:18 AM
Dzahn changed the task status from Open to Stalled.Apr 10 2020, 9:32 AM

stalled by T247018

Dzahn changed the task status from Stalled to Open.May 22 2020, 12:19 PM
gerritbot added a comment.May 29 2020, 9:10 AM

Change 599749 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add new appservers mw2336 through mw2339

https://gerrit.wikimedia.org/r/599749

gerritbot added a project: Patch-For-Review.May 29 2020, 9:10 AM
gerritbot added a comment.Jun 10 2020, 10:50 AM

Change 604339 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add management and production IPs for mw2335-mw2339

https://gerrit.wikimedia.org/r/604339

gerritbot added a comment.Jun 10 2020, 2:46 PM

Change 604339 merged by Dzahn:
[operations/dns@master] add management and production IPs for mw2335-mw2339

https://gerrit.wikimedia.org/r/604339

gerritbot added a comment.Jun 10 2020, 3:11 PM

Change 599749 merged by Dzahn:
[operations/puppet@production] site: add new appservers mw2335 through mw2339

https://gerrit.wikimedia.org/r/599749

Maintenance_bot removed a project: Patch-For-Review.Jun 10 2020, 4:10 PM
Stashbot added a comment.Jun 17 2020, 2:13 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-17T14:13:33Z] <mutante> generating new mcrouter certs for mw2335 - mw2339 (T247021)

gerritbot added a comment.Jun 17 2020, 2:21 PM

Change 606195 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake mcrouter certs for mw2335 - mw2339

https://gerrit.wikimedia.org/r/606195

gerritbot added a project: Patch-For-Review.Jun 17 2020, 2:21 PM

Change 606195 merged by Dzahn:
[labs/private@master] add fake mcrouter certs for mw2335 - mw2339

https://gerrit.wikimedia.org/r/606195

Dzahn mentioned this in rLPRI10ce4a9db662: add fake mcrouter certs for mw2335 - mw2339.Jun 17 2020, 2:24 PM
gerritbot added a comment.Jun 17 2020, 2:24 PM

Change 606197 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] conftool: add mw2335 - mw2339

https://gerrit.wikimedia.org/r/606197

ops-monitoring-bot added a comment.Jun 17 2020, 2:27 PM

Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 5 host(s) and their services with reason: new_install

mw[2335-2339].codfw.wmnet
gerritbot added a comment.Jun 17 2020, 2:31 PM

Change 606197 merged by Dzahn:
[operations/puppet@production] conftool: add mw2335 - mw2339

https://gerrit.wikimedia.org/r/606197

RhinosF1 subscribed.Jun 17 2020, 2:31 PM
ops-monitoring-bot added a comment.Jun 17 2020, 2:52 PM

Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 5 host(s) and their services with reason: new_install

mw[2335-2339].codfw.wmnet
Maintenance_bot removed a project: Patch-For-Review.Jun 17 2020, 3:11 PM
Dzahn added a comment.Jun 17 2020, 3:24 PM

mw2335 through mw2339 in rack C3 have also been taken into production now.

This should complete the ticket. All hosts from mw2291 through mw2376 are pooled.

Just mw2377 seems to be missing.

Dzahn closed this task as Resolved.Jun 17 2020, 3:33 PM

76 servers are pooled as appservers. 10 have been used for kubernetes. Adds up to 86.

ops-monitoring-bot added a comment.Jun 18 2020, 6:17 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2339.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202006181816_dzahn_145765_mw2339_codfw_wmnet.log.

ops-monitoring-bot added a comment.Jun 18 2020, 6:51 PM

Completed auto-reimage of hosts:

['mw2339.codfw.wmnet']

and were ALL successful.

Dzahn added a subtask: T258614: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping.Jul 22 2020, 7:53 PM
jijiki mentioned this in T260654: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420).Aug 18 2020, 7:17 AM
Dzahn added a subtask: T260654: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420).Aug 28 2020, 5:45 PM
Dzahn closed subtask T260654: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420) as Resolved.Aug 28 2020, 7:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL