Page MenuHomePhabricator
Paste P49604

scap backport 927713 (parse1002.eqiad.wmnet)
ActivePublic
Actions

Authored by TheresNoTime on Jul 19 2023, 8:21 PM.
Project Tags
None
Referenced Files
F37143931: scap backport 927713 (parse1002.eqiad.wmnet)
Jul 19 2023, 8:28 PM
F37143929: scap backport 927713 (parse1002.eqiad.wmnet)
Jul 19 2023, 8:26 PM
F37143926: scap backport 927713 (parse1002.eqiad.wmnet)
Jul 19 2023, 8:24 PM
F37143924: scap backport 927713
Jul 19 2023, 8:21 PM
Subscribers
dancy
20:18:43 sync-proxies: 100% (in-flight: 0; ok: 8; fail: 0; left: 0)
20:18:43 Per-host sync duration: average 3.7s, median 3.6s
20:18:43 rsync transfer: average 1,461,018 bytes/host, total 11,688,144 bytes
20:18:43 Finished sync-proxies (duration: 00m 04s)
20:18:43 Started sync-apaches
20:21:08 ['/usr/bin/scap', 'pull', '--no-php-restart', '--no-update-l10n', '--exclude-wikiversions.php', 'deploy1002.eqiad.wmnet', 'mw1366.eqiad.wmnet', 'mw2289.codfw.wmnet', 'mw2259.codfw.wmnet', 'mw1398.eqiad.wmnet', 'mw1404.eqiad.wmnet', 'mw1420.eqiad.wmnet', 'mw2300.codfw.wmnet', 'deploy2002.codfw.wmnet', 'mw1486.eqiad.wmnet'] (ran as mwdeploy@parse1002.eqiad.wmnet) returned [255]: ssh: connect to host parse1002.eqiad.wmnet port 22: Connection timed out
20:21:08 sync-apaches: 100% (in-flight: 0; ok: 370; fail: 1; left: 0)
20:21:08 Per-host sync duration: average 4.4s, median 3.4s
20:21:08 rsync transfer: average 1,457,079 bytes/host, total 540,576,660 bytes
20:21:08 1 apaches had sync errors
20:21:08 Finished sync-apaches (duration: 02m 24s)
20:21:08 Started scap-cdb-rebuild
20:23:23 sudo -u mwdeploy -n -- /usr/bin/scap cdb-rebuild (ran as mwdeploy@parse1002.eqiad.wmnet) returned [255]: ssh: connect to host parse1002.eqiad.wmnet port 22: Connection timed out
20:23:23 scap-cdb-rebuild: 100% (in-flight: 0; ok: 391; fail: 1; left: 0)
20:23:23 1 hosts had scap-cdb-rebuild errors
20:23:23 Finished scap-cdb-rebuild (duration: 02m 15s)
20:23:23 Started sync_wikiversions
20:25:39 sudo -u mwdeploy -n -- /usr/bin/rsync -l deploy1002.eqiad.wmnet::common/wikiversions*.{json,php} /srv/mediawiki (ran as mwdeploy@parse1002.eqiad.wmnet) returned [255]: ssh: connect to host parse1002.eqiad.wmnet port 22: Connection timed out
20:25:39 sync_wikiversions: 100% (in-flight: 0; ok: 391; fail: 1; left: 0)
20:25:39 Finished sync_wikiversions (duration: 02m 15s)
20:25:39 1 hosts had sync_wikiversions errors
20:25:39 Started php-fpm-restarts
20:25:39 Running '/usr/local/sbin/restart-php-fpm-all php7.4-fpm 9223372036854775807' on 311 host(s)
20:28:33 /usr/bin/sudo -u root -- /usr/local/sbin/restart-php-fpm-all php7.4-fpm 9223372036854775807 (ran as mwdeploy@parse1002.eqiad.wmnet) returned [255]: ssh: connect to host parse1002.eqiad.wmnet port 22: Connection timed out
20:28:33 php-fpm-restart: 100% (in-flight: 0; ok: 310; fail: 1; left: 0)
20:28:33 1 hosts had failures restarting php-fpm
20:28:33 Finished php-fpm-restarts (duration: 02m 54s)
20:28:33 Running /usr/local/bin/mwscript purgeMessageBlobStore.php
20:28:33 Finished scap: Backport for [[gerrit:927713|Replace underscores with spaces in 4 Arabic sitenames (T337725)]] (duration: 17m 09s)
20:28:33 backport failed: <CalledProcessError> Command '['/usr/bin/scap', 'sync-world', '--pause-after-testserver-sync', '--notify-user=hubaishan', 'Backport for [[gerrit:927713|Replace underscores with spaces in 4 Arabic sitenames (T337725)]]']' returned non-zero exit status 1.
[samtar@deploy1002 ~]$

Event Timeline

TheresNoTime created this paste.Jul 19 2023, 8:21 PM
TheresNoTime changed the title of this paste from scap backport 927713 to scap backport 927713 (parse1002.eqiad.wmnet).Jul 19 2023, 8:24 PM
TheresNoTime edited the content of this paste. (Show Details)
dancy subscribed.Jul 19 2023, 8:25 PM

Ack. Saw this myself when rolling the train.

TheresNoTime edited the content of this paste. (Show Details)Jul 19 2023, 8:26 PM
TheresNoTime edited the content of this paste. (Show Details)Jul 19 2023, 8:28 PM
TheresNoTime mentioned this in T342298: parse1002 down.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits