Page MenuHomePhabricator
Create Task
Maniphest T279309

bring 43 new mediawiki appserver in eqiad into production
Closed, ResolvedPublic
Actions

Description

45 (43) new MediaWiki appservers were procured in T271155.

43 are being racked in T273915.

Just like T278396 for codfw this is the task to:

  • make a plan how many become appserver, API server, jobrunner/videoscaler, canaries, or ... dedicated jobrunner (T279100)
  • add appropriate regexes to site.pp, matching the puppet roles above
  • add hosts to conftool-data to the right sections
  • initial puppet run, reboot, check monitoring, set weight, pool
  • setup canaries and recreate hieradata (removed in 702659)
  • old servers have to be decom'ed in parallel (-> T280203)

Rack A3

  • mw1414.eqiad.wmnet - appserver
  • mw1415.eqiad.wmnet - appserver
  • mw1416.eqiad.wmnet - appserver
  • mw1417.eqiad.wmnet - appserver
  • mw1418.eqiad.wmnet - appserver
  • mw1419.eqiad.wmnet - appserver
  • mw1420.eqiad.wmnet - appserver
  • mw1421.eqiad.wmnet - API server
  • mw1422.eqiad.wmnet - API server

Rack B3

  • mw1423.eqiad.wmnet - API server
  • mw1424.eqiad.wmnet - API server
  • mw1425.eqiad.wmnet - API server
  • mw1426.eqiad.wmnet - API server
  • mw1427.eqiad.wmnet - API server
  • mw1428.eqiad.wmnet - API server
  • mw1429.eqiad.wmnet - appserver
  • mw1430.eqiad.wmnet - appserver
  • mw1431.eqiad.wmnet - appserver
  • mw1432.eqiad.wmnet - appserver
  • mw1433.eqiad.wmnet - appserver

Rack C3

  • mw1434.eqiad.wmnet - appserver
  • mw1435.eqiad.wmnet - appserver
  • mw1436.eqiad.wmnet - appserver

Rack D8

  • mw1437.eqiad.wmnet - jobrunner canary
  • mw1438.eqiad.wmnet - jobrunner canary
  • mw1439.eqiad.wmnet - appserver
  • mw1440.eqiad.wmnet - appserver
  • mw1441.eqiad.wmnet - appserver
  • mw1442.eqiad.wmnet - appserver
  • mw1443.eqiad.wmnet - API server
  • mw1444.eqiad.wmnet - API server (!) - NOT REACHABLE via SSH - FIXED
  • mw1445.eqiad.wmnet - jobrunner
  • mw1446.eqiad.wmnet - jobrunner
  • mw1447.eqiad.wmnet - Canary API server
  • mw1448.eqiad.wmnet - Canary API server
  • mw1449.eqiad.wmnet - Canary API server
  • mw1450.eqiad.wmnet - Canary API server

Rack A1

  • mw1451.eqiad.wmnet - appserver
  • mw1452.eqiad.wmnet - appserver

Rack A8

  • mw1453.eqiad.wmnet - appserver
  • mw1454.eqiad.wmnet - appserver
  • mw1455.eqiad.wmnet - appserver
  • mw1456.eqiad.wmnet - appserver

Details

SubjectRepoBranchLines +/-
mw130[2-6]: Remove and decomissionoperations/puppetproduction+3 -13
scap: Switch mw1306 to mw1318 for scap proxy roleoperations/puppetproduction+1 -1
hieradata: cleanup old canary api serveroperations/puppetproduction+4 -899
site/conftool: add mw1451 through mw1456 as apppservers, A1, A8operations/puppetproduction+16 -1
site/conftool: add mw1447,mw1448,mw1449,mw1450 as canary API appserversoperations/puppetproduction+10 -1
site: remove mw1444 from 'insetup' roleoperations/puppetproduction+1 -1
DHCP: add MAC addresses for mw1448 through mw1456operations/puppetproduction+45 -0
site: remove already installed servers from insetup regexoperations/puppetproduction+1 -1
site/conftool: convert 4 appservers to jobrunners in row D for balanceoperations/puppetproduction+13 -11
site/conftool: convert mw1434-mw1436 from API to app serversoperations/puppetproduction+4 -4
site/conftool: add mw1439, mw1440 as jobrunnersoperations/puppetproduction+7 -0
add mcrouter certs for mw1422.eqiad.wmnet to mw1446.eqiad.wmnetlabs/privatemaster+0 -0
site/conftool: add mw1443 through mw1446 as API appserversoperations/puppetproduction+5 -1
site/conftool: add mw1439 through mw1442 as appservers, rack D8operations/puppetproduction+9 -0
conftool: convert mw1421, mw1422 from app to API servers for balanceoperations/puppetproduction+8 -3
site/conftool: add mw1437,mw1438 as canary jobrunnersoperations/puppetproduction+7 -0
site/conftool: add mw1434,mw1435,mw1436 as API appserversoperations/puppetproduction+7 -0
role::common::mediawiki::canary_appserver add new canary app server in eqiadoperations/puppetproduction+14 -7
site: add mw1430 to regex for new appserver rangeoperations/puppetproduction+1 -1
site/conftool: add mw1429 through mw1433 as appservers, rack B3operations/puppetproduction+9 -0
site/conftool: add mw1426, mw1427, mw1428 as API appserversoperations/puppetproduction+4 -1
site: re-add mw1422 to regex for appserversoperations/puppetproduction+1 -1
site/conftool: turn mw1423,mw1424,mw1425 into API appserversoperations/puppetproduction+9 -1
site/conftool: turn mw1422 into an mw appserveroperations/puppetproduction+3 -2
add mcrouter certs for mw1415.eqiad.wmnet to mw1421.eqiad.wmnetlabs/privatemaster+0 -0
add mcrouter certs for mw1414.eqiad.wmnetlabs/privatemaster+0 -0
site: add eight appsevers in eqiad row A3operations/puppetproduction+14 -1
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 Resolved• CmjohnsonT273915 (Need By: TBD) rack/setup/install mw14[14-56]
 ResolvedDzahnT279309 bring 43 new mediawiki appserver in eqiad into production

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jul 20 2021, 4:46 PM

Change 705721 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: add mw1437,mw1438 as canary jobrunners

https://gerrit.wikimedia.org/r/705721

Jelto updated the task description. (Show Details)Jul 21 2021, 8:21 AM
Jelto updated the task description. (Show Details)
gerritbot added a comment.Jul 21 2021, 12:26 PM

Change 705721 merged by Dzahn:

[operations/puppet@production] site/conftool: add mw1437,mw1438 as canary jobrunners

https://gerrit.wikimedia.org/r/705721

Dzahn updated the task description. (Show Details)Jul 21 2021, 1:05 PM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Jul 21 2021, 2:55 PM

Change 705927 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: add mw1439, mw1440 as jobrunners

https://gerrit.wikimedia.org/r/705927

Volans subscribed.Jul 21 2021, 3:19 PM

FYI if that helps this is the current row-distribution of the API appservers in eqiad:

{'B': 19, 'D': 18, 'C': 17, 'A': 9}

Full details at P16841

gerritbot added a comment.Jul 21 2021, 4:04 PM

Change 705943 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] conftool: convert mw1421, mw1422 from app to API servers for balance

https://gerrit.wikimedia.org/r/705943

ops-monitoring-bot added a comment.Jul 21 2021, 4:08 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1421.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202107211607_dzahn_31447_mw1421_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Jul 21 2021, 4:09 PM

Completed auto-reimage of hosts:

['mw1421.eqiad.wmnet']

Of which those FAILED:

['mw1421.eqiad.wmnet']
Volans added a comment.Jul 21 2021, 4:51 PM

FYI I've updated the pastes for eqiad and codfw with some more detailed data, all yours now :)

gerritbot added a comment.Jul 22 2021, 10:12 AM

Change 705943 merged by Dzahn:

[operations/puppet@production] conftool: convert mw1421, mw1422 from app to API servers for balance

https://gerrit.wikimedia.org/r/705943

ops-monitoring-bot added a comment.Jul 22 2021, 10:19 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1421.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202107221018_dzahn_8762_mw1421_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Jul 22 2021, 10:24 AM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw1422.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202107221024_dzahn_13185_mw1422_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Jul 22 2021, 10:57 AM

Completed auto-reimage of hosts:

['mw1421.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Jul 22 2021, 11:10 AM

Completed auto-reimage of hosts:

['mw1422.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jul 22 2021, 1:43 PM

Change 706485 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] site/conftool: add mw1439,mw1440,mw1441,mw1442 as canary API appservers

https://gerrit.wikimedia.org/r/706485

Dzahn updated the task description. (Show Details)Jul 22 2021, 7:09 PM
gerritbot added a comment.Jul 23 2021, 8:39 AM

Change 707252 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: add mw1437 through mw1440 as appservers, rack D8

https://gerrit.wikimedia.org/r/707252

Dzahn updated the task description. (Show Details)Jul 23 2021, 8:45 AM
Dzahn updated the task description. (Show Details)Jul 23 2021, 8:53 AM
gerritbot added a comment.Jul 23 2021, 8:56 AM

Change 707252 merged by Dzahn:

[operations/puppet@production] site/conftool: add mw1439 through mw1442 as appservers, rack D8

https://gerrit.wikimedia.org/r/707252

Dzahn updated the task description. (Show Details)Jul 23 2021, 9:52 AM
gerritbot added a comment.Jul 23 2021, 10:01 AM

Change 707298 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: add mw1443 through mw1446 as API appservers

https://gerrit.wikimedia.org/r/707298

Dzahn updated the task description. (Show Details)Jul 23 2021, 10:03 AM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Jul 23 2021, 10:08 AM

Change 707300 had a related patch set uploaded (by Jelto; author: Jelto):

[labs/private@master] add mcrouter certs for mw1422.eqiad.wmnet to mw1442.eqiad.wmnet

https://gerrit.wikimedia.org/r/707300

gerritbot added a comment.Jul 23 2021, 10:23 AM

Change 707298 merged by Dzahn:

[operations/puppet@production] site/conftool: add mw1443 through mw1446 as API appservers

https://gerrit.wikimedia.org/r/707298

Dzahn updated the task description. (Show Details)Jul 23 2021, 11:17 AM
gerritbot added a comment.Jul 23 2021, 11:28 AM

Change 707300 merged by Dzahn:

[labs/private@master] add mcrouter certs for mw1422.eqiad.wmnet to mw1446.eqiad.wmnet

https://gerrit.wikimedia.org/r/707300

Jelto mentioned this in rLPRIc545963eb6fc: add mcrouter certs for mw1422.eqiad.wmnet to mw1446.eqiad.wmnet.Jul 23 2021, 11:30 AM
gerritbot added a comment.Jul 23 2021, 11:47 AM

Change 705927 abandoned by Dzahn:

[operations/puppet@production] site/conftool: add mw1439, mw1440 as jobrunners

Reason:

already used as appservers

https://gerrit.wikimedia.org/r/705927

Jelto updated the task description. (Show Details)Jul 23 2021, 12:14 PM
Stashbot added a comment.Jul 23 2021, 12:15 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-23T12:15:39Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mw1439.eqiad.wmnet with reason: setup new canary mw api servers in eqiad D8 https://phabricator.wikimedia.org/T279309

Stashbot added a comment.Jul 23 2021, 12:15 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-23T12:15:47Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw1439.eqiad.wmnet with reason: setup new canary mw api servers in eqiad D8 https://phabricator.wikimedia.org/T279309

Stashbot added a comment.Jul 23 2021, 12:16 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-23T12:16:12Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on mw[1440-1442].eqiad.wmnet with reason: setup new canary mw api servers in eqiad D8 https://phabricator.wikimedia.org/T279309

Stashbot added a comment.Jul 23 2021, 12:16 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-23T12:16:19Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mw[1440-1442].eqiad.wmnet with reason: setup new canary mw api servers in eqiad D8 https://phabricator.wikimedia.org/T279309

Jelto updated the task description. (Show Details)Jul 23 2021, 12:54 PM
Dzahn added a subscriber: wiki_willy.Jul 28 2021, 11:14 AM

@wiki_willy Here it would be great for us if next someone could finish the setup of mw1447 through mw1450 and take a look at special case mw1444 which should be ready but isn't reachable yet. Thanks!

Dzahn triaged this task as High priority.Jul 28 2021, 11:14 AM
gerritbot added a comment.Jul 28 2021, 1:31 PM

Change 708526 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: convert mw134-mw1436 from API to app servers

https://gerrit.wikimedia.org/r/708526

Dzahn added a comment.Jul 28 2021, 1:43 PM

mw1434 has an issue with IPMI

Remote IPMI failed for mgmt 'mw1434.mgmt.eqiad.wmnet': Command '['ipmitool', '-I', 'lanplus', '-H', 'mw1434.mgmt.eqiad.wmnet', '-U', 'root', '-E', 'chassis', 'power', 'status']' returned non-zero exit status 1.
gerritbot added a comment.Jul 28 2021, 1:44 PM

Change 708526 merged by Dzahn:

[operations/puppet@production] site/conftool: convert mw1434-mw1436 from API to app servers

https://gerrit.wikimedia.org/r/708526

ops-monitoring-bot added a comment.Jul 28 2021, 1:45 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

['mw1435.eqiad.wmnet', 'mw1436.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202107281345_dzahn_32210.log.

ops-monitoring-bot added a comment.Jul 28 2021, 2:26 PM

Completed auto-reimage of hosts:

['mw1435.eqiad.wmnet', 'mw1436.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Jul 28 2021, 2:39 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

['mw1434.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202107281439_dzahn_10859.log.

ops-monitoring-bot added a comment.Jul 28 2021, 3:19 PM

Completed auto-reimage of hosts:

['mw1434.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jul 28 2021, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-28T15:53:58Z] <mutante> mw1434,mw1435,mw1436: scap pull, repooled, reimaged, converted from API to appserver for balancing (T279309)

Dzahn updated the task description. (Show Details)Jul 28 2021, 3:54 PM
Dzahn added a comment.Jul 29 2021, 2:44 PM
In T279309#7242742, @Dzahn wrote:

@wiki_willy Here it would be great for us if next someone could finish the setup of mw1447 through mw1450 and take a look at special case mw1444 which should be ready but isn't reachable yet. Thanks!

due to the balancing between rows, when you install the remaining servers please put them in A or at least _not_ in D please. thanks!

Jclark-ctr subscribed.Jul 29 2021, 11:04 PM

@Dzahn Thanks! putting the rest in A will speed up racking currently i was waiting on rack C

gerritbot added a comment.Jul 30 2021, 1:37 PM

Change 709041 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: convert 4 appservers to jobrunners in row D for balance

https://gerrit.wikimedia.org/r/709041

gerritbot added a comment.Jul 30 2021, 1:51 PM

Change 709041 merged by Dzahn:

[operations/puppet@production] site/conftool: convert 4 appservers to jobrunners in row D for balance

https://gerrit.wikimedia.org/r/709041

ops-monitoring-bot added a comment.Jul 30 2021, 1:56 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

['mw1439.eqiad.wmnet', 'mw1440.eqiad.wmnet', 'mw1445.eqiad.wmnet', 'mw1446.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202107301356_dzahn_2612.log.

ops-monitoring-bot added a comment.Jul 30 2021, 2:40 PM

Completed auto-reimage of hosts:

['mw1439.eqiad.wmnet', 'mw1440.eqiad.wmnet', 'mw1445.eqiad.wmnet', 'mw1446.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jul 30 2021, 2:42 PM

Change 709064 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: remove already installed servers from insetup regex

https://gerrit.wikimedia.org/r/709064

gerritbot added a comment.Jul 30 2021, 2:56 PM

Change 709064 merged by Dzahn:

[operations/puppet@production] site: remove already installed servers from insetup regex

https://gerrit.wikimedia.org/r/709064

Jelto updated the task description. (Show Details)Aug 2 2021, 7:53 AM
Jelto subscribed.Aug 5 2021, 12:54 PM
Dzahn updated the task description. (Show Details)Aug 11 2021, 8:32 AM
Stashbot added a comment.Aug 13 2021, 9:35 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T09:35:42Z] <mutante> mw1444 - signed puppet cert, initial run (after hardware fix) T279309

Stashbot added a comment.Aug 13 2021, 9:42 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T09:42:23Z] <mutante> mw1448, mw1449, mw1450 - powering on via mgmt - OS install, initial setup (T279309, T273915)

gerritbot added a comment.Aug 13 2021, 10:04 AM

Change 712928 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] DHCP: add MAC addresses for mw1448 through mw1456

https://gerrit.wikimedia.org/r/712928

gerritbot added a comment.Aug 13 2021, 10:19 AM

Change 712928 merged by Dzahn:

[operations/puppet@production] DHCP: add MAC addresses for mw1448 through mw1456

https://gerrit.wikimedia.org/r/712928

Stashbot added a comment.Aug 13 2021, 11:11 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T11:11:35Z] <jelto> mw1455 - powering on via mgmt - OS install, initial setup (T279309, T273915)

ops-monitoring-bot added a comment.Aug 13 2021, 12:11 PM

Script wmf-auto-reimage was launched by jelto on cumin1001.eqiad.wmnet for hosts:

mw1455.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202108131211_jelto_16075_mw1455_eqiad_wmnet.log.

gerritbot added a comment.Aug 13 2021, 12:20 PM

Change 712939 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: remove mw1444 from 'insetup' role

https://gerrit.wikimedia.org/r/712939

Dzahn updated the task description. (Show Details)Aug 13 2021, 12:21 PM
ops-monitoring-bot added a comment.Aug 13 2021, 12:35 PM

Completed auto-reimage of hosts:

['mw1455.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Aug 13 2021, 12:46 PM

Change 712939 merged by Dzahn:

[operations/puppet@production] site: remove mw1444 from 'insetup' role

https://gerrit.wikimedia.org/r/712939

Jelto updated the task description. (Show Details)Aug 13 2021, 1:13 PM
Stashbot added a comment.Aug 13 2021, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T13:21:37Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw[1447-1449].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309

Stashbot added a comment.Aug 13 2021, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T13:21:45Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[1447-1449].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309

Stashbot added a comment.Aug 13 2021, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T13:21:54Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1450.eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309

Stashbot added a comment.Aug 13 2021, 1:22 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T13:22:03Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1450.eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309

gerritbot added a comment.Aug 13 2021, 1:25 PM

Change 706485 merged by Jelto:

[operations/puppet@production] site/conftool: add mw1447,mw1448,mw1449,mw1450 as canary API appservers

https://gerrit.wikimedia.org/r/706485

gerritbot added a comment.Aug 13 2021, 2:18 PM

Change 712970 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/conftool: add mw1451 through mw1456 as apppservers, A1, A8

https://gerrit.wikimedia.org/r/712970

gerritbot added a comment.Aug 13 2021, 2:26 PM

Change 712970 merged by Dzahn:

[operations/puppet@production] site/conftool: add mw1451 through mw1456 as apppservers, A1, A8

https://gerrit.wikimedia.org/r/712970

Stashbot added a comment.Aug 13 2021, 5:05 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T17:05:44Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw[1451-1452,1454-1455].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309

Stashbot added a comment.Aug 13 2021, 5:06 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T17:06:02Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw[1451-1452,1454-1455].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309

Jelto updated the task description. (Show Details)Aug 13 2021, 5:13 PM
Stashbot added a comment.Aug 13 2021, 5:32 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T17:32:12Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on mw[1451-1452,1454-1455].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309

Stashbot added a comment.Aug 13 2021, 5:32 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-13T17:32:19Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on mw[1451-1452,1454-1455].eqiad.wmnet with reason: setup new mediawiki servers in eqiad https://phabricator.wikimedia.org/T279309

Jelto updated the task description. (Show Details)Aug 17 2021, 7:40 AM
Dzahn updated the task description. (Show Details)Aug 17 2021, 7:51 AM
Dzahn closed this task as Resolved.Aug 17 2021, 9:19 AM
Dzahn updated the task description. (Show Details)

done !:) All new servers are finally in production now.

gerritbot added a comment.Aug 18 2021, 8:30 AM

Change 713607 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] hieradata::hosts::mw1 cleanup old canary api server hieradata

https://gerrit.wikimedia.org/r/713607

gerritbot added a comment.Aug 18 2021, 2:11 PM

Change 713607 merged by Jelto:

[operations/puppet@production] hieradata: cleanup old canary api server

https://gerrit.wikimedia.org/r/713607

gerritbot added a comment.Mar 3 2022, 1:33 PM

Change 767787 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] scap: Switch mw1306 to mw1318 for scap proxy role

https://gerrit.wikimedia.org/r/767787

gerritbot added a comment.Mar 3 2022, 1:33 PM

Change 767788 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] mw130[2-6]: Remove and decomission

https://gerrit.wikimedia.org/r/767788

gerritbot added a comment.Mar 4 2022, 8:45 AM

Change 767787 merged by Alexandros Kosiaris:

[operations/puppet@production] scap: Switch mw1306 to mw1318 for scap proxy role

https://gerrit.wikimedia.org/r/767787

gerritbot added a comment.Mar 4 2022, 1:12 PM

Change 767788 merged by Alexandros Kosiaris:

[operations/puppet@production] mw130[2-6]: Remove and decomission

https://gerrit.wikimedia.org/r/767788

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits