Page MenuHomePhabricator
Paste P6832

36 apaches had sync errors
ArchivedPublic

Authored by zeljkofilipin on Mar 12 2018, 1:35 PM.
Tags
None
Referenced Files
F15190925: 36 apaches had sync errors
Mar 12 2018, 1:35 PM
Subscribers
None
zfilipin@tin:/srv/mediawiki-staging$ scap sync-file wmf-config/Wikibase-production.php 'SWAT: [[gerrit:416748|Enable caching of constraint check results (T184812)]]'
___ ____
⎛ ⎛ ,----
\ //==--'
_//|,.·//==--' ____________________________
_OO≣=- ︶ ᴹw ⎞_§ ______ ___\ ___\ ,\__ \/ __ \
(∞)_, ) ( | ______/__ \/ /__ / /_/ / /_/ /
¨--¨|| |- ( / ______\____/ \___/ \__^_/ .__/
««_/ «_/ jgs/bd808 /_/
13:28:25 Copying to tin.eqiad.wmnet from tin.eqiad.wmnet
13:28:25 Started rsync common
13:28:25 Finished rsync common (duration: 00m 00s)
13:28:26 Started cache_git_info
13:28:42 Finished cache_git_info (duration: 00m 16s)
13:28:42 Started sync-masters
sync-masters: 100% (ok: 1; fail: 0; left: 0)
13:28:50 Finished sync-masters (duration: 00m 07s)
13:28:50 Started sync-pull-masters
sync-pull-masters: 100% (ok: 1; fail: 0; left: 0)
13:28:51 Finished sync-pull-masters (duration: 00m 01s)
13:28:51 Started sync-check-canaries
check-canaries: 100% (ok: 11; fail: 0; left: 0)
13:28:54 Finished sync-check-canaries (duration: 00m 02s)
13:28:54 Waiting for canary traffic...
13:29:14 Executing check 'Logstash Error rate for mw1279.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mw1276.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mw1261.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mw1264.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mwdebug1002.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mwdebug1001.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mw1263.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mw1262.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mw1278.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mw1277.eqiad.wmnet'
13:29:14 Executing check 'Logstash Error rate for mw1265.eqiad.wmnet'
13:29:15 Started sync-proxies
sync-proxies: 100% (ok: 7; fail: 0; left: 0)
13:29:18 Finished sync-proxies (duration: 00m 03s)
13:29:18 Started sync-apaches
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2123.codfw.wmnet returned [255]: ssh: connect to host mw2123.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2130.codfw.wmnet returned [255]: ssh: connect to host mw2130.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2107.codfw.wmnet returned [255]: ssh: connect to host mw2107.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2125.codfw.wmnet returned [255]: ssh: connect to host mw2125.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2117.codfw.wmnet returned [255]: ssh: connect to host mw2117.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2127.codfw.wmnet returned [255]: ssh: connect to host mw2127.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2133.codfw.wmnet returned [255]: ssh: connect to host mw2133.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2122.codfw.wmnet returned [255]: ssh: connect to host mw2122.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2116.codfw.wmnet returned [255]: ssh: connect to host mw2116.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2102.codfw.wmnet returned [255]: ssh: connect to host mw2102.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2124.codfw.wmnet returned [255]: ssh: connect to host mw2124.codfw.wmnet port 22: Connection timed out
13:31:30 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2128.codfw.wmnet returned [255]: ssh: connect to host mw2128.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2111.codfw.wmnet returned [255]: ssh: connect to host mw2111.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2115.codfw.wmnet returned [255]: ssh: connect to host mw2115.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2126.codfw.wmnet returned [255]: ssh: connect to host mw2126.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2105.codfw.wmnet returned [255]: ssh: connect to host mw2105.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2110.codfw.wmnet returned [255]: ssh: connect to host mw2110.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2101.codfw.wmnet returned [255]: ssh: connect to host mw2101.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2100.codfw.wmnet returned [255]: ssh: connect to host mw2100.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2106.codfw.wmnet returned [255]: ssh: connect to host mw2106.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2134.codfw.wmnet returned [255]: ssh: connect to host mw2134.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2109.codfw.wmnet returned [255]: ssh: connect to host mw2109.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2121.codfw.wmnet returned [255]: ssh: connect to host mw2121.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2103.codfw.wmnet returned [255]: ssh: connect to host mw2103.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2118.codfw.wmnet returned [255]: ssh: connect to host mw2118.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2132.codfw.wmnet returned [255]: ssh: connect to host mw2132.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2097.codfw.wmnet returned [255]: ssh: connect to host mw2097.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2104.codfw.wmnet returned [255]: ssh: connect to host mw2104.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2120.codfw.wmnet returned [255]: ssh: connect to host mw2120.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2119.codfw.wmnet returned [255]: ssh: connect to host mw2119.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2113.codfw.wmnet returned [255]: ssh: connect to host mw2113.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2131.codfw.wmnet returned [255]: ssh: connect to host mw2131.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2129.codfw.wmnet returned [255]: ssh: connect to host mw2129.codfw.wmnet port 22: Connection timed out
13:31:32 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2112.codfw.wmnet returned [255]: ssh: connect to host mw2112.codfw.wmnet port 22: Connection timed out
13:31:34 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2108.codfw.wmnet returned [255]: ssh: connect to host mw2108.codfw.wmnet port 22: Connection timed out
13:31:34 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/Wikibase-production.php', 'mw1284.eqiad.wmnet', 'mw1319.eqiad.wmnet', 'mw1280.eqiad.wmnet', 'mw2215.codfw.wmnet', 'mw2187.codfw.wmnet', 'mw1250.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2114.codfw.wmnet returned [255]: ssh: connect to host mw2114.codfw.wmnet port 22: Connection timed out
sync-apaches: 100% (ok: 238; fail: 36; left: 0)
13:31:34 36 apaches had sync errors
13:31:34 Finished sync-apaches (duration: 02m 15s)
13:31:34 Synchronized wmf-config/Wikibase-production.php: SWAT: [[gerrit:416748|Enable caching of constraint check results (T184812)]] (duration: 03m 08s)