Page MenuHomePhabricator
Create Task
Maniphest T239054

Reimage all mediawiki servers
Closed, ResolvedPublic
Actions

Description

The final stage in cleaning up hhvm, are reimaging all mw servers.

Tracking current status: https://etherpad.wikimedia.org/p/reimagemw

Reimaging command wmf-auto-reimage --sequential -c -a -p T239054 FQDN

eqiad
We are going to decomm 36 eqiad servers, and will add another 36+23

  • Decomm mw[1221-1235,1238-1258].eqiad.wmnet
  • Reimage mw[1261-1290,1293-1348].eqiad.wmnet
  • mw1348.eqiad.wmnet API
  • mw1347.eqiad.wmnet API
  • mw1346.eqiad.wmnet API
  • mw1345.eqiad.wmnet API
  • mw1344.eqiad.wmnet API
  • mw1343.eqiad.wmnet API
  • mw1342.eqiad.wmnet API
  • mw1341.eqiad.wmnet API
  • mw1340.eqiad.wmnet API
  • mw1339.eqiad.wmnet API
  • mw1338.eqiad.wmnet Jobrunner
  • mw1337.eqiad.wmnet Jobrunner
  • mw1336.eqiad.wmnet Jobrunner
  • mw1335.eqiad.wmnet Jobrunner
  • mw1334.eqiad.wmnet Jobrunner
  • mw1333.eqiad.wmnet Appserver
  • mw1332.eqiad.wmnet Appserver
  • mw1331.eqiad.wmnet Appserver
  • mw1330.eqiad.wmnet Appserver
  • mw1329.eqiad.wmnet Appserver
  • mw1328.eqiad.wmnet Appserver
  • mw1327.eqiad.wmnet Appserver
  • mw1326.eqiad.wmnet Appserver
  • mw1325.eqiad.wmnet Appserver
  • mw1324.eqiad.wmnet Appserver
  • mw1323.eqiad.wmnet Appserver
  • mw1322.eqiad.wmnet Appserver effie
  • mw1321.eqiad.wmnet LEAVE FOR LAST, mcrouter proxy Appserver
  • mw1320.eqiad.wmnet Appserver LEAVE FOR LAST, scap proxy
  • mw1319.eqiad.wmnet Appserver
  • mw1318.eqiad.wmnet Jobrunner
  • mw1317.eqiad.wmnet API effie
  • mw1316.eqiad.wmnet API
  • mw1315.eqiad.wmnet API
  • mw1314.eqiad.wmnet API LEAVE FOR LAST, scap proxy
  • mw1313.eqiad.wmnet API
  • mw1312.eqiad.wmnet API
  • mw1311.eqiad.wmnet Jobrunner
  • mw1310.eqiad.wmnet Jobrunner
  • mw1309.eqiad.wmnet Jobrunner
  • mw1308.eqiad.wmnet Jobrunner
  • mw1307.eqiad.wmnet Jobrunner
  • mw1306.eqiad.wmnet Jobrunner
  • mw1305.eqiad.wmnet Jobrunner
  • mw1304.eqiad.wmnet Jobrunner
  • mw1303.eqiad.wmnet Jobrunner
  • mw1302.eqiad.wmnet Jobrunner
  • mw1301.eqiad.wmnet Jobrunner
  • mw1300.eqiad.wmnet Jobrunner
  • mw1299.eqiad.wmnet Jobrunner
  • mw1298.eqiad.wmnet Jobrunner
  • mw1297.eqiad.wmnet API
  • mw1296.eqiad.wmnet Jobrunner
  • mw1295.eqiad.wmnet Jobrunner
  • mw1294.eqiad.wmnet Jobrunner
  • mw1293.eqiad.wmnet Jobrunner
  • mw1290.eqiad.wmnet API
  • mw1289.eqiad.wmnet API
  • mw1288.eqiad.wmnet API
  • mw1287.eqiad.wmnet API
  • mw1286.eqiad.wmnet LEAVE FOR LAST, mcrouter proxy API
  • mw1285.eqiad.wmnet API
  • mw1284.eqiad.wmnet API
  • mw1283.eqiad.wmnet API
  • mw1282.eqiad.wmnet API
  • mw1281.eqiad.wmnet API
  • mw1280.eqiad.wmnet API
  • mw1279.eqiad.wmnet API
  • mw1278.eqiad.wmnet API
  • mw1277.eqiad.wmnet API
  • mw1276.eqiad.wmnet API
  • mw1275.eqiad.wmnet Appserver
  • mw1274.eqiad.wmnet Appserver
  • mw1273.eqiad.wmnet Appserver
  • mw1272.eqiad.wmnet Appserver
  • mw1271.eqiad.wmnet Appserver
  • mw1270.eqiad.wmnet Appserver
  • mw1269.eqiad.wmnet LEAVE FOR LAST, mcrouter proxy Appserver
  • mw1268.eqiad.wmnet Appserver
  • mw1267.eqiad.wmnet Appserver
  • mw1266.eqiad.wmnet Appserver
  • mw1265.eqiad.wmnet Appserver
  • mw1264.eqiad.wmnet Appserver
  • mw1263.eqiad.wmnet Appserver
  • mw1262.eqiad.wmnet Appserver
  • mw1261.eqiad.wmnet Appserver

codfw

We are going to decomm 77 servers and add 63

  • Decomm mw[2135-2214].codfw.wmnet
  • Reimage mw[2215-2290].codfw.wmnet
  • mw2290.codfw.wmnet API
  • mw2289.codfw.wmnet API
  • mw2288.codfw.wmnet API
  • mw2287.codfw.wmnet API
  • mw2286.codfw.wmnet API
  • mw2285.codfw.wmnet API
  • mw2284.codfw.wmnet API
  • mw2283.codfw.wmnet API
  • mw2282.codfw.wmnet Jobrunner
  • mw2281.codfw.wmnet Jobrunner
  • mw2280.codfw.wmnet Jobrunner
  • mw2279.codfw.wmnet Jobrunner
  • mw2278.codfw.wmnet Jobrunner
  • mw2277.codfw.wmnet Appserver
  • mw2276.codfw.wmnet Appserver
  • mw2275.codfw.wmnet Appserver
  • mw2274.codfw.wmnet Appserver
  • mw2273.codfw.wmnet Appserver
  • mw2272.codfw.wmnet Appserver
  • mw2271.codfw.wmnet LEAVE FOR LAST, mcrouter proxy Appserver
  • mw2270.codfw.wmnet Appserver
  • mw2269.codfw.wmnet Appserver
  • mw2268.codfw.wmnet Appserver
  • mw2267.codfw.wmnet Jobrunner
  • mw2266.codfw.wmnet Jobrunner
  • mw2265.codfw.wmnet Jobrunner
  • mw2264.codfw.wmnet Jobrunner
  • mw2263.codfw.wmnet Jobrunner
  • mw2262.codfw.wmnet API
  • mw2261.codfw.wmnet API
  • mw2260.codfw.wmnet Jobrunner
  • mw2259.codfw.wmnet Jobrunner - BROKEN -> T239758
  • mw2258.codfw.wmnet Appserver
  • mw2257.codfw.wmnet Appserver
  • mw2256.codfw.wmnet Appserver
  • mw2255.codfw.wmnet LEAVE FOR LAST, mcrouter proxy Appserver
  • mw2254.codfw.wmnet Appserverx
  • mw2253.codfw.wmnet API
  • mw2252.codfw.wmnet API
  • mw2251.codfw.wmnet API
  • mw2250.codfw.wmnet Jobrunner
  • mw2249.codfw.wmnet Jobrunner
  • mw2248.codfw.wmnet Jobrunner
  • mw2247.codfw.wmnet Jobrunner
  • mw2246.codfw.wmnet Jobrunner
  • mw2245.codfw.wmnet API
  • mw2244.codfw.wmnet API
  • mw2243.codfw.wmnet Jobrunner
  • mw2242.codfw.wmnet Appserver
  • mw2241.codfw.wmnet Appserver
  • mw2240.codfw.wmnet Appserver
  • mw2239.codfw.wmnet Appserver
  • mw2238.codfw.wmnet Appserver
  • mw2237.codfw.wmnet Appserver
  • mw2236.codfw.wmnet Appserver
  • mw2235.codfw.wmnet LEAVE FOR LAST, mcrouter proxy Appserver
  • mw2234.codfw.wmnet Appserver
  • mw2233.codfw.wmnet Appserver
  • mw2232.codfw.wmnet Appserver
  • mw2231.codfw.wmnet Appserver
  • mw2230.codfw.wmnet Appserver
  • mw2229.codfw.wmnet Appserver
  • mw2228.codfw.wmnet Appserver
  • mw2227.codfw.wmnet Appserver
  • mw2226.codfw.wmnet Appserver
  • mw2225.codfw.wmnet Appserver
  • mw2224.codfw.wmnet Appserver
  • mw2223.codfw.wmnet API
  • mw2222.codfw.wmnet API
  • mw2221.codfw.wmnet API
  • mw2220.codfw.wmnet API
  • mw2219.codfw.wmnet API
  • mw2218.codfw.wmnet API
  • mw2217.codfw.wmnet API
  • mw2216.codfw.wmnet API LEAVE FOR LAST, scap proxy
  • mw2215.codfw.wmnet API

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+5 -5scap:dsh.yaml switch scap proxies so to reimage them
operations/puppetproduction+6 -6common::mcrouter.yaml switch mcrouter proxies so to reimage them
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 ResolvedJdforrester-WMFT233012 Make MediaWiki core compatible with PHP 7.4
 ResolvedMaxSemT216734 Drop devDependency on phpunit-mock-objects, which is abandoned and warns against use
 ResolvedNoneT192167 Upgrade PHPUnit from 4/6 to 8
 ResolvedReedyT196206 Bump symfony libraries when we no longer need hhvm support
 ResolvedDaimonaT220589 Support phan in MediaWiki's default composer.json
 ResolvedDaimonaT231710 Recommend PHP7-only features after dropping HHVM support
 ResolvedaaronT232128 Make MultiHttpClient use CURLMOPT_MAX_HOST_CONNECTIONS and reuse connections
 ResolvedMaxSemT234236 Drop HHVM support from MediaWiki-Vagrant
 ResolvedReedyT227242 Deploy WebAuthn to Wikimedia Wikis
 ResolvedDaimonaT231636 Butcher the list of phan suppressions in MW core
 ResolvedReedyT235266 Composer Library Updates, late 2019/early 2020 edition
 ResolvedReedyT235233 Stop using PHP5 versions of ruflin/elastica and elasticsearch/elasticsearch
 ResolvedJdforrester-WMFT234512 Archive HHVM Phabricator project
 ResolvedKrinkleT235160 Decide whether extensions should set a PHP version requirement in composer.json
 ResolvedKrinkleT235142 Clean up mediawiki-config from HHVM-related configuration
 ResolvedJdforrester-WMFT192166 Drop HHVM support from MediaWiki
 ResolvedJdforrester-WMFT234384 Drop HHVM from CI
 ResolvedjijikiT219127 SRE FY19-20 Q1 goal: complete the transition to PHP7
 ResolvedjijikiT229792 Remove HHVM from production
 ResolvedjijikiT239054 Reimage all mediawiki servers

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
fgiunchedi updated the task description. (Show Details)Dec 2 2019, 11:51 AM
ops-monitoring-bot added a comment.Dec 2 2019, 12:38 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['mw1296.eqiad.wmnet', 'mw1298.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912021238_jiji_246071.log.

ops-monitoring-bot added a comment.Dec 2 2019, 12:46 PM

Completed auto-reimage of hosts:

['mw1275.eqiad.wmnet', 'mw1290.eqiad.wmnet']

and were ALL successful.

jijiki updated the task description. (Show Details)Dec 2 2019, 12:55 PM
ops-monitoring-bot added a comment.Dec 2 2019, 2:51 PM

Completed auto-reimage of hosts:

['mw1296.eqiad.wmnet', 'mw1298.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 4:19 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['mw1295.eqiad.wmnet', 'mw1294.eqiad.wmnet', 'mw1293.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912021619_jiji_32185.log.

ops-monitoring-bot added a comment.Dec 2 2019, 5:43 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2240.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021743_dzahn_48992_mw2240_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 5:45 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2239.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021745_dzahn_49266_mw2239_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 5:57 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2238.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021757_dzahn_50879_mw2238_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 5:59 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2237.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021759_dzahn_51262_mw2237_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 6:54 PM

Completed auto-reimage of hosts:

['mw2240.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 6:55 PM

Completed auto-reimage of hosts:

['mw2239.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 7:02 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2245.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021901_dzahn_65268_mw2245_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:02 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2234.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021902_dzahn_65614_mw2234_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:09 PM

Completed auto-reimage of hosts:

['mw2238.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 7:10 PM

Completed auto-reimage of hosts:

['mw2237.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 7:25 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2246.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021923_dzahn_71993_mw2246_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:25 PM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

mw2233.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021925_ariel_72277_mw2233_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:25 PM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

mw2232.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021925_ariel_72297_mw2232_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:25 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2247.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021925_dzahn_72347_mw2247_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:25 PM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

mw2231.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021925_ariel_72344_mw2231_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:26 PM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

mw2234.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021926_ariel_72413_mw2234_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:26 PM

Completed auto-reimage of hosts:

['mw2234.codfw.wmnet']

Of which those FAILED:

['mw2234.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 2 2019, 7:27 PM

Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts:

mw2234.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021927_ariel_72531_mw2234_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:30 PM

Completed auto-reimage of hosts:

['mw2234.codfw.wmnet']

Of which those FAILED:

['mw2234.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 2 2019, 7:33 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2230.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912021933_dzahn_74631_mw2230_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 7:40 PM

Completed auto-reimage of hosts:

['mw1295.eqiad.wmnet', 'mw1294.eqiad.wmnet', 'mw1293.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 8:12 PM

Completed auto-reimage of hosts:

['mw2245.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 8:33 PM

Completed auto-reimage of hosts:

['mw2247.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 8:35 PM

Completed auto-reimage of hosts:

['mw2246.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 8:39 PM

Completed auto-reimage of hosts:

['mw2231.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 8:39 PM

Completed auto-reimage of hosts:

['mw2233.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 8:41 PM

Completed auto-reimage of hosts:

['mw2234.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 8:42 PM

Completed auto-reimage of hosts:

['mw2232.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 8:47 PM

Completed auto-reimage of hosts:

['mw2230.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 9:34 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2248.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912022134_dzahn_104033_mw2248_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 9:35 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2249.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912022135_dzahn_105363_mw2249_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 9:36 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2251.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912022136_dzahn_105527_mw2251_codfw_wmnet.log.

Dzahn updated the task description. (Show Details)Dec 2 2019, 9:39 PM
ops-monitoring-bot added a comment.Dec 2 2019, 10:34 PM

Completed auto-reimage of hosts:

['mw2248.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 10:42 PM

Completed auto-reimage of hosts:

['mw2249.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 10:48 PM

Completed auto-reimage of hosts:

['mw2251.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 2 2019, 11:37 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2250.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912022337_dzahn_132438_mw2250_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 11:38 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2252.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912022338_dzahn_132541_mw2252_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 11:42 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2253.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912022342_dzahn_132970_mw2253_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2019, 11:43 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2254.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912022343_dzahn_133083_mw2254_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 3 2019, 12:41 AM

Completed auto-reimage of hosts:

['mw2250.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 3 2019, 12:48 AM

Completed auto-reimage of hosts:

['mw2252.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 3 2019, 12:57 AM

Completed auto-reimage of hosts:

['mw2254.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 3 2019, 1:14 AM

Completed auto-reimage of hosts:

['mw2253.codfw.wmnet']

and were ALL successful.

Dzahn updated the task description. (Show Details)Dec 3 2019, 2:39 AM
ops-monitoring-bot added a comment.Dec 3 2019, 7:10 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2256.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912031909_dzahn_88754_mw2256_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 3 2019, 7:11 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2257.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912031911_dzahn_89118_mw2257_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 3 2019, 7:14 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2258.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912031913_dzahn_89640_mw2258_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 3 2019, 7:14 PM

Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts:

mw2259.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/201912031914_dzahn_89825_mw2259_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 3 2019, 8:23 PM

Completed auto-reimage of hosts:

['mw2256.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 3 2019, 8:23 PM

Completed auto-reimage of hosts:

['mw2257.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 3 2019, 8:26 PM

Completed auto-reimage of hosts:

['mw2258.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Dec 3 2019, 8:43 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-03T20:43:32Z] <mutante> mw2259 - did not come back from reboot after reimage, also mgmt not reachable (T239054)

ops-monitoring-bot added a comment.Dec 3 2019, 8:47 PM

Completed auto-reimage of hosts:

['mw2259.codfw.wmnet']

Of which those FAILED:

['mw2259.codfw.wmnet']
Dzahn mentioned this in T239758: mw2259 down and mgmt does not exist?.Dec 3 2019, 9:34 PM
Dzahn updated the task description. (Show Details)Dec 3 2019, 10:02 PM
ops-monitoring-bot added a comment.Dec 4 2019, 1:40 PM

Script wmf-auto-reimage was launched by rzl on cumin1001.eqiad.wmnet for hosts:

['mw2274.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912041329_rzl_100644.log.

ops-monitoring-bot added a comment.Dec 4 2019, 2:21 PM

Completed auto-reimage of hosts:

['mw2274.codfw.wmnet']

and were ALL successful.

RLazarus updated the task description. (Show Details)Dec 4 2019, 2:41 PM
ops-monitoring-bot added a comment.Dec 4 2019, 2:48 PM

Script wmf-auto-reimage was launched by rzl on cumin1001.eqiad.wmnet for hosts:

['mw2273.codfw.wmnet', 'mw2272.codfw.wmnet', 'mw2267.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912041448_rzl_118042.log.

ops-monitoring-bot added a comment.Dec 4 2019, 4:47 PM

Completed auto-reimage of hosts:

['mw2273.codfw.wmnet', 'mw2272.codfw.wmnet', 'mw2267.codfw.wmnet']

and were ALL successful.

RLazarus updated the task description. (Show Details)Dec 4 2019, 6:45 PM
ops-monitoring-bot added a comment.Dec 4 2019, 6:58 PM

Script wmf-auto-reimage was launched by rzl on cumin1001.eqiad.wmnet for hosts:

['mw2266.codfw.wmnet', 'mw2265.codfw.wmnet', 'mw2264.codfw.wmnet', 'mw2263.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912041858_rzl_170459.log.

ops-monitoring-bot added a comment.Dec 4 2019, 9:26 PM

Completed auto-reimage of hosts:

['mw2266.codfw.wmnet', 'mw2265.codfw.wmnet', 'mw2264.codfw.wmnet', 'mw2263.codfw.wmnet']

and were ALL successful.

RLazarus updated the task description. (Show Details)Dec 4 2019, 9:42 PM
ops-monitoring-bot added a comment.Dec 5 2019, 10:27 AM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['mw2260.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912051027_jiji_93252.log.

ops-monitoring-bot added a comment.Dec 5 2019, 11:05 AM

Completed auto-reimage of hosts:

['mw2260.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 5 2019, 12:22 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['mw2261.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912051222_jiji_116344.log.

ops-monitoring-bot added a comment.Dec 5 2019, 1:03 PM

Completed auto-reimage of hosts:

['mw2261.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 5 2019, 2:20 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['mw2260.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912051420_jiji_139305.log.

ops-monitoring-bot added a comment.Dec 5 2019, 4:05 PM

Completed auto-reimage of hosts:

['mw2260.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 5 2019, 5:19 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['mw2260.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912051719_jiji_174984.log.

ops-monitoring-bot added a comment.Dec 5 2019, 5:54 PM

Completed auto-reimage of hosts:

['mw2260.codfw.wmnet']

and were ALL successful.

Dzahn updated the task description. (Show Details)Dec 5 2019, 6:44 PM
jijiki updated the task description. (Show Details)Dec 6 2019, 9:05 AM
jijiki mentioned this in T229792: Remove HHVM from production.Dec 6 2019, 9:08 AM
ops-monitoring-bot added a comment.Dec 9 2019, 1:56 PM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['mw2270.codfw.wmnet', 'mw2269.codfw.wmnet', 'mw2268.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912091356_jiji_197404.log.

ops-monitoring-bot added a comment.Dec 9 2019, 4:01 PM

Completed auto-reimage of hosts:

['mw2270.codfw.wmnet', 'mw2269.codfw.wmnet', 'mw2268.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Dec 18 2019, 11:25 AM

Change 559021 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] scap:dsh.yaml switch scap proxies so to reimage them

https://gerrit.wikimedia.org/r/559021

gerritbot added a project: Patch-For-Review.Dec 18 2019, 11:25 AM
gerritbot added a comment.Dec 18 2019, 12:21 PM

Change 559033 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] common::mcrouter.yaml switch mcrouter proxies so to reimage them

https://gerrit.wikimedia.org/r/559033

gerritbot added a comment.Dec 18 2019, 4:02 PM

Change 559033 merged by Effie Mouzeli:
[operations/puppet@production] common::mcrouter.yaml switch mcrouter proxies so to reimage them

https://gerrit.wikimedia.org/r/559033

gerritbot added a comment.Dec 18 2019, 4:10 PM

Change 559021 merged by Effie Mouzeli:
[operations/puppet@production] scap:dsh.yaml switch scap proxies so to reimage them

https://gerrit.wikimedia.org/r/559021

Maintenance_bot removed a project: Patch-For-Review.Dec 18 2019, 4:11 PM
ops-monitoring-bot added a comment.Dec 19 2019, 4:44 AM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['mw1321.eqiad.wmnet', 'mw1320.eqiad.wmnet', 'mw1314.eqiad.wmnet', 'mw2271.codfw.wmnet', 'mw2255.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912190444_jiji_33118.log.

ops-monitoring-bot added a comment.Dec 19 2019, 9:49 AM

Completed auto-reimage of hosts:

['mw1321.eqiad.wmnet', 'mw1320.eqiad.wmnet', 'mw1314.eqiad.wmnet', 'mw2271.codfw.wmnet', 'mw2255.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Dec 19 2019, 9:56 AM

Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts:

['mw1286.eqiad.wmnet', 'mw1269.eqiad.wmnet', 'mw2235.codfw.wmnet', 'mw2216.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201912190955_jiji_101664.log.

ops-monitoring-bot added a comment.Dec 19 2019, 2:27 PM

Completed auto-reimage of hosts:

['mw1286.eqiad.wmnet', 'mw1269.eqiad.wmnet', 'mw2235.codfw.wmnet', 'mw2216.codfw.wmnet']

and were ALL successful.

jijiki closed this task as Resolved.Dec 19 2019, 7:11 PM
jijiki claimed this task.
jijiki updated the task description. (Show Details)

FIN!

Dzahn awarded a token.Dec 20 2019, 2:41 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL