Page MenuHomePhabricator

(Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet
Closed, ResolvedPublic

Description

This task will track the racking, setup, and installation of 36 new mw systems ordered via T228214.

Racking Proposal: Due to the recent order of additional mw hosts on upcoming task T233639, the racking for this needs to be re-evaluated by @Joe and team to ensure they get the best placement for this batch of 36 and the other batch of 23 systems incoming. This section will be updated by them.

Individual server setup checklist (this checklist should be duplicated for each system):

  • - receive in system on procurement task T228214.
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementation steps by service owners -> https://wikitech.wikimedia.org/wiki/Application_servers#Adding_a_new_server_into_production
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Joe added a comment.Nov 5 2019, 6:21 AM

Sorry I'm lagging behind on this. Point is, I shouldn't be the only person to answer this question but most of my team is out this week. I'll try to reassign this to someone with time to respond.

@Joe et al - just following up on this, since we've had these servers onsite for a while now. Also, once you have any specific racking instructions for us, please keep in mind that we'll need some time to complete all the tasks associated with getting these hosts set up, so it should be reflected in your requested "need by date" as well.

Thanks,
Willy

wiki_willy renamed this task from rack/setup/install mw13[49-84].eqiad.wmnet to (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet.Nov 22 2019, 8:57 PM
jijiki added a subscriber: jijiki.Nov 22 2019, 10:07 PM

@wiki_willy I will provide racking instructions on Monday for you, sorry we have delayed you this much.

Thanks @jijiki , much appreciated

Currently:

RackA 5A6A 7B 6B 7C 6D 4D 5
mw servers6617216306 (decom)30 (decom)

We will decommission 36 servers from D 4 and D 5.

I recommend we add:

  • D 1: 10 servers
  • D 2: 6 servers
  • D 6: 10 servers
  • D 8: 10 servers

Regarding the 23 upcoming ones, we will split them between rows A, B, C. Please let me know if that rack setup makes sense. Thank you!

jijiki added a comment.Dec 5 2019, 4:46 PM

@Jclark-ctr Do we have an estimate when we are able to have those servers racked?

Hi @jijiki - we received these on Sept 25, but just recently received the racking instructions from your team on Nov 25, so there might be some shuffling around we need to do, before getting these racked to meet the recommendations. When do you need these handed over to you by? And we can try to meet that need by date. Thanks, Willy

jijiki added a comment.Dec 5 2019, 5:29 PM

@wiki_willy given we are responsible for this delay, we would like to check with your team and tell us what is the earliest we can do, and we can in turn can plan around that. Sorry for the delays on our end. I would like to avoid requesting a date that is unrealistic, given that xmas is round the corner. Thank you!

Thanks for confirming @jijiki - I'll let @Jclark-ctr provide an ETA on them, when he gets in a bit later today. Thanks, Willy

@jijiki If no surprises i could have them Racked by Dec 20th

jijiki added a comment.Dec 5 2019, 5:43 PM

That would be lovely, thank you!

wiki_willy renamed this task from (No Need By Date Provided) rack/setup/install mw13[49-84].eqiad.wmnet to (Need By Dec 20) rack/setup/install mw13[49-84].eqiad.wmnet.Dec 5 2019, 5:46 PM
Papaul added a subscriber: Papaul.Dec 6 2019, 1:18 AM

@Jclark-ctr Please see below for available mgmt IP's that you can use for those servers. Once you have the asset tags please update the table with the information and we can enter the the name, ip and asset tag in the DNS file. Let me know if you have any questions. Thanks

mgmt IP's

HostIPAsset tag
mw134910.65.1.24
mw135010.65.1.25
mw135110.65.2.6
mw135210.65.2.7
mw135310.65.2.10
mw135410.65.2.14
mw135510.65.2.17
mw135610.65.2.18
mw135710.65.2.19
mw135810.65.2.21
mw135910.65.2.26
mw136010.65.2.33
mw136110.65.2.34
mw136210.65.2.35
mw136310.65.2.42
mw136410.65.2.69
mw136510.65.2.70
mw136610.65.2.109
mw136710.65.2.119
mw136810.65.2.120
mw136910.65.2.121
mw137010.65.2.129
mw137110.65.2.130
mw137210.65.2.131
mw137310.65.2.132
mw137410.65.2.133
mw137510.65.2.134
mw137610.65.2.135
mw137710.65.3.15
mw137810.65.3.25
mw137910.65.3.27
mw138010.65.3.28
mw138110.65.3.30
mw138210.65.3.31
mw138310.65.3.32
mw138410.65.3.33

@jijiki

I recommend we use these racks we are limited in space . In row D and would not have the enough sfp to rj45 to accommodate the putting these in 10g racks
D 1: 15 servers
D 3: 3 servers
D 6: 15 servers
D 8: 3 servers

@Jclark-ctr if you feel this will work better, we are happy. Either way, this racking is still better than the original one (30 servers in D 5). Thank you!

@jijiki small delay had a few tickets become urgent. Being a contractor i will be in week after christmas and will wrap them up at beginning of new year.

Hey @Jclark-ctr ! Do we have an update about when we will have those servers ready? Thank you!

jijiki added a subscriber: Cmjohnson.EditedJan 22 2020, 4:21 PM

@Jclark-ctr @Cmjohnson @wiki_willy please let us know the status of racking these servers and update the task accordingly, thank you!

Jclark-ctr updated the task description. (Show Details)

Host racked bios, ip , and password set. Needs dns

server ip asset tag rack switch port
mw1349 10.65.1.24 WMF5291 D1 42
mw1350 10.65.1.25 WMF5292 D1 41
mw1351 10.65.2.6 WMF5293 D1 40
mw1352 10.65.2.7 WMF5294 D1 39
mw1353 10.65.2.10 WMF5295 D1 38
mw1354 10.65.2.14 WMF5296 D1 37
mw1355 10.65.2.17 WMF5297 D1 36
mw1356 10.65.2.18 WMF5298 D1 35
mw1357 10.65.2.19 WMF5299 D1 33
mw1358 10.65.2.21 WMF5300 D1 34
mw1359 10.65.2.26 WMF5301 D1 29
mw1360 10.65.2.33 WMF5302 D1 30
mw1361 10.65.2.34 WMF5303 D1 31
mw1362 10.65.2.35 WMF5304 D1 32
mw1363 10.65.2.42 WMF5305 D3 40
mw1364 10.65.2.69 WMF5306 D3 41
mw1365 10.65.2.70 WMF5307 D3 42
mw1366 10.65.2.109 WMF5308 D6 27
mw1367 10.65.2.119 WMF5309 D6 26
mw1368 10.65.2.120 WMF5310 D6 25
mw1369 10.65.2.121 WMF5311 D6 24
mw1370 10.65.2.129 WMF5312 D6 23
mw1371 10.65.2.130 WMF5313 D6 22
mw1372 10.65.2.131 WMF5314 D6 21
mw1373 10.65.2.132 WMF5315 D6 20
mw1374 10.65.2.133 WMF5316 D6 19
mw1375 10.65.2.134 WMF5317 D6 18
mw1376 10.65.2.135 WMF5318 D6 17
mw1377 10.65.3.15 WMF5319 D6 16
mw1378 10.65.3.25 WMF5320 D6 15
mw1379 10.65.3.27 WMF5321 D6 14
mw1380 10.65.3.28 WMF5322 D6 13
mw1381 10.65.3.30 WMF5323 D6 12
mw1382 10.65.3.31 WMF5324 D6 11
mw1383 10.65.3.32 WMF5325 D8 25
mw1384 10.65.3.33 WMF5326 D8 26

Joe raised the priority of this task from Medium to High.Feb 4 2020, 7:53 AM

Can we please expedite this?

We really need these servers to join rotation.

@Joe working on these next...will have to you in next day or 2.

mark added a subscriber: mark.Feb 12 2020, 4:47 PM

@wiki_willy With Chris having been ill the past few days, what's a realistic new ETA for this?

@mark - just chatted with Chris and he's working on them now (he's back today after being sick), so ETA is end of day. Thanks, Willy

RobH removed a subscriber: RobH.Feb 12 2020, 5:32 PM

Change 571782 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns for mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/571782

Change 571782 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns for mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/571782

Change 571785 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/571785

Change 571785 merged by Cmjohnson:
[operations/dns@master] Adding production dns mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/571785

Change 571797 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding macs mw servers. Missing mw1351,79-81 for bad passwd

https://gerrit.wikimedia.org/r/571797

Cmjohnson updated the task description. (Show Details)Feb 12 2020, 8:45 PM

@Jclark-ctr when you're at data center later today could you look into these please.

mw1351 and mw1381 have the wrong password set
mw1379 and mw1380 are not getting a connection, most likely cable related.

Change 571797 merged by Cmjohnson:
[operations/puppet@production] Adding macs mw servers. Missing mw1351,79-81 for bad passwd

https://gerrit.wikimedia.org/r/571797

Latest update: there were some settings that need to be redone, so @Cmjohnson will get these fixed tomorrow, with Thursday as the new ETA. Apologies for the long delay. Thanks, Willy

Change 572031 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac to dhcpd for mw1351

https://gerrit.wikimedia.org/r/572031

Change 572031 merged by Cmjohnson:
[operations/puppet@production] Adding mac to dhcpd for mw1351

https://gerrit.wikimedia.org/r/572031

Change 572066 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac/dhcpd file entry for mw1379[80-81]

https://gerrit.wikimedia.org/r/572066

Change 572066 merged by Cmjohnson:
[operations/puppet@production] Adding mac/dhcpd file entry for mw1379[80-81]

https://gerrit.wikimedia.org/r/572066

Cmjohnson reassigned this task from Cmjohnson to Joe.Feb 13 2020, 7:47 PM
Cmjohnson updated the task description. (Show Details)
Cmjohnson removed a project: ops-eqiad.

@Joe These are ready for you now, initial puppet run has been completed. I am removing from ops-eqiad tag. If you need any additional on-site work please add tag back and ping me. assigning to you

Joe removed Joe as the assignee of this task.Feb 18 2020, 7:00 AM
Joe added subscribers: RLazarus, Dzahn.

@RLazarus and @Dzahn can you pick up the task of putting these into rotation?

They will eventually replace mw1221-mw1258, and I think we can go 1:1 in terms of replacing Appservers and Apis. Let's keep an eye on maintaining (or reinstating!) the per-row balance of both clusters.

Change 572975 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add new eqiad mw appservers with spare role

https://gerrit.wikimedia.org/r/572975

Change 572975 merged by Dzahn:
[operations/puppet@production] add new eqiad mw appservers with spare role

https://gerrit.wikimedia.org/r/572975

Change 572992 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] Site: Assign mw1349 as mediawiki::appserver

https://gerrit.wikimedia.org/r/572992

Change 572992 merged by RLazarus:
[operations/puppet@production] site: Assign mw1349 as mediawiki::appserver

https://gerrit.wikimedia.org/r/572992

Change 573002 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[labs/private@master] Add fake keys for mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/573002

Change 573002 merged by RLazarus:
[labs/private@master] Add fake keys for mw13[49-84].eqiad.wmnet

https://gerrit.wikimedia.org/r/573002

Change 573019 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] assign mw1350 through mw1355 as MediaWiki appservers

https://gerrit.wikimedia.org/r/573019

Change 573019 merged by Dzahn:
[operations/puppet@production] assign mw1350 through mw1355 as MediaWiki appservers

https://gerrit.wikimedia.org/r/573019

Dzahn added a comment.Feb 19 2020, 1:30 AM

The first 7: mw1349 through mw1355 have been added as regular appservers and are pooled now. But just with weight 10. We will change weights and add more (API) appservers tomorrow.

Weights of mw1349-mw1355 were switched to 30

Change 573352 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] site: Assign mw13[56-62] as mediawiki::appserver::api.

https://gerrit.wikimedia.org/r/573352

Change 573352 merged by RLazarus:
[operations/puppet@production] site: Assign mw13[56-62] as mediawiki::appserver::api.

https://gerrit.wikimedia.org/r/573352

Dzahn added a comment.Feb 19 2020, 9:04 PM

etherpad with role layout and status: https://etherpad.wikimedia.org/p/T236437

Icinga downtime for 1:00:00 set by rzl@cumin1001 on 11 host(s) and their services with reason: new installs

mw[1364-1373,1384].eqiad.wmnet

Icinga downtime for 1:00:00 set by rzl@cumin1001 on 11 host(s) and their services with reason: new installs

mw[1364-1373,1384].eqiad.wmnet

Change 573402 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/conftooldata: add more new eqiad MW API appservers

https://gerrit.wikimedia.org/r/573402

Change 573402 merged by Dzahn:
[operations/puppet@production] site/conftooldata: add more new eqiad MW API appservers

https://gerrit.wikimedia.org/r/573402

Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 11 host(s) and their services with reason: new_install

mw[1363,1374-1383].eqiad.wmnet

Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 11 host(s) and their services with reason: new_install

mw[1363,1374-1383].eqiad.wmnet
Dzahn claimed this task.Feb 20 2020, 12:04 AM
Dzahn updated the task description. (Show Details)Feb 20 2020, 12:13 AM

All hosts have roles, weight 30, have been pooled and changed to status "Active" in Netbox.

{"mw1349.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1349.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1350.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1350.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1351.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1351.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1352.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1352.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1353.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1353.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1354.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1354.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1355.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1355.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1356.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1356.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1357.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1357.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1358.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1358.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1359.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1359.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1360.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1360.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1361.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1361.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1362.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1362.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1363.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1363.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1364.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1364.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1365.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1365.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1366.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1366.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1367.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1367.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1368.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1368.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1369.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1369.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1370.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1370.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1371.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1371.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1372.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1372.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1373.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1373.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
{"mw1374.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1374.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1375.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1375.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1376.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1376.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1377.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1377.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1378.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1378.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1379.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1379.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1380.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1380.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1381.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1381.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1382.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1382.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1383.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=apache2"}
{"mw1383.eqiad.wmnet": {"weight": 30, "pooled": "yes"}, "tags": "dc=eqiad,cluster=api_appserver,service=nginx"}
{"mw1384.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=apache2"}
{"mw1384.eqiad.wmnet": {"pooled": "yes", "weight": 30}, "tags": "dc=eqiad,cluster=appserver,service=nginx"}
Dzahn closed this task as Resolved.Feb 20 2020, 12:15 AM
elukey added a subscriber: elukey.Feb 23 2020, 4:53 PM

As FYI I just powercycled mw1372 since it was frozen (no ssh, no mgmt serial console usable, etc..).