During both the train and UTC evening backport, parse1002 was down.
Can someone please depool it and check for issues?
| 1 | 20:18:43 sync-proxies: 100% (in-flight: 0; ok: 8; fail: 0; left: 0) |
|---|---|
| 2 | 20:18:43 Per-host sync duration: average 3.7s, median 3.6s |
| 3 | 20:18:43 rsync transfer: average 1,461,018 bytes/host, total 11,688,144 bytes |
| 4 | 20:18:43 Finished sync-proxies (duration: 00m 04s) |
| 5 | 20:18:43 Started sync-apaches |
| 6 | 20:21:08 ['/usr/bin/scap', 'pull', '--no-php-restart', '--no-update-l10n', '--exclude-wikiversions.php', 'deploy1002.eqiad.wmnet', 'mw1366.eqiad.wmnet', 'mw2289.codfw.wmnet', 'mw2259.codfw.wmnet', 'mw1398.eqiad.wmnet', 'mw1404.eqiad.wmnet', 'mw1420.eqiad.wmnet', 'mw2300.codfw.wmnet', 'deploy2002.codfw.wmnet', 'mw1486.eqiad.wmnet'] (ran as mwdeploy@parse1002.eqiad.wmnet) returned [255]: ssh: connect to host parse1002.eqiad.wmnet port 22: Connection timed out |
| 7 | |
| 8 | 20:21:08 sync-apaches: 100% (in-flight: 0; ok: 370; fail: 1; left: 0) |
| 9 | 20:21:08 Per-host sync duration: average 4.4s, median 3.4s |
| 10 | 20:21:08 rsync transfer: average 1,457,079 bytes/host, total 540,576,660 bytes |
| 11 | 20:21:08 1 apaches had sync errors |
| 12 | 20:21:08 Finished sync-apaches (duration: 02m 24s) |
| 13 | 20:21:08 Started scap-cdb-rebuild |
| 14 | 20:23:23 sudo -u mwdeploy -n -- /usr/bin/scap cdb-rebuild (ran as mwdeploy@parse1002.eqiad.wmnet) returned [255]: ssh: connect to host parse1002.eqiad.wmnet port 22: Connection timed out |
| 15 | |
| 16 | 20:23:23 scap-cdb-rebuild: 100% (in-flight: 0; ok: 391; fail: 1; left: 0) |
| 17 | 20:23:23 1 hosts had scap-cdb-rebuild errors |
| 18 | 20:23:23 Finished scap-cdb-rebuild (duration: 02m 15s) |
| 19 | 20:23:23 Started sync_wikiversions |
| 20 | 20:25:39 sudo -u mwdeploy -n -- /usr/bin/rsync -l deploy1002.eqiad.wmnet::common/wikiversions*.{json,php} /srv/mediawiki (ran as mwdeploy@parse1002.eqiad.wmnet) returned [255]: ssh: connect to host parse1002.eqiad.wmnet port 22: Connection timed out |
| 21 | |
| 22 | 20:25:39 sync_wikiversions: 100% (in-flight: 0; ok: 391; fail: 1; left: 0) |
| 23 | 20:25:39 Finished sync_wikiversions (duration: 02m 15s) |
| 24 | 20:25:39 1 hosts had sync_wikiversions errors |
| 25 | 20:25:39 Started php-fpm-restarts |
| 26 | 20:25:39 Running '/usr/local/sbin/restart-php-fpm-all php7.4-fpm 9223372036854775807' on 311 host(s) |
| 27 | 20:28:33 /usr/bin/sudo -u root -- /usr/local/sbin/restart-php-fpm-all php7.4-fpm 9223372036854775807 (ran as mwdeploy@parse1002.eqiad.wmnet) returned [255]: ssh: connect to host parse1002.eqiad.wmnet port 22: Connection timed out |
| 28 | |
| 29 | 20:28:33 php-fpm-restart: 100% (in-flight: 0; ok: 310; fail: 1; left: 0) |
| 30 | 20:28:33 1 hosts had failures restarting php-fpm |
| 31 | 20:28:33 Finished php-fpm-restarts (duration: 02m 54s) |
| 32 | 20:28:33 Running /usr/local/bin/mwscript purgeMessageBlobStore.php |
| 33 | 20:28:33 Finished scap: Backport for [[gerrit:927713|Replace underscores with spaces in 4 Arabic sitenames (T337725)]] (duration: 17m 09s) |
| 34 | 20:28:33 backport failed: <CalledProcessError> Command '['/usr/bin/scap', 'sync-world', '--pause-after-testserver-sync', '--notify-user=hubaishan', 'Backport for [[gerrit:927713|Replace underscores with spaces in 4 Arabic sitenames (T337725)]]']' returned non-zero exit status 1. |
| 35 | [samtar@deploy1002 ~]$ |