[zfilipin@deploy1001 mediawiki-staging (master * u=)]$ scap clean --delete 1.32.0-wmf.6 ___ ____ ⎛ ⎛ ,---- \ //==--' _//|,.·//==--' ____________________________ _OO≣=- ︶ ᴹw ⎞_§ ______ ___\ ___\ ,\__ \/ __ \ (∞)_, ) ( | ______/__ \/ /__ / /_/ / /_/ / ¨--¨|| |- ( / ______\____/ \___/ \__^_/ .__/ ««_/ «_/ jgs/bd808 /_/ 13:28:55 Started clean-l10nupdate-cache 13:28:55 Finished clean-l10nupdate-cache (duration: 00m 00s) 13:28:55 Started clean-l10nupdate-owned-files 13:28:55 Finished clean-l10nupdate-owned-files (duration: 00m 00s) 13:28:55 Started clean-ExtensionMessages 13:28:55 Finished clean-ExtensionMessages (duration: 00m 00s) 13:28:55 Started prune-git-branches remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done error: unable to delete 'wmf/1.32.0-wmf.6': remote ref does not exist error: failed to push some refs to 'https://gerrit.wikimedia.org/r/mediawiki/extensions/CentralNotice' remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done remote: Processing changes: done 13:30:18 Finished prune-git-branches (duration: 01m 22s) 13:30:18 Started removing-local-copy 13:30:19 Finished removing-local-copy (duration: 00m 01s) 13:30:19 Started cleaning-unused-patches 13:30:19 Finished cleaning-unused-patches (duration: 00m 00s) 13:30:19 Copying to deploy1001.eqiad.wmnet from deploy1001.eqiad.wmnet 13:30:19 Started rsync common cannot delete non-empty directory: php-1.32.0-wmf.6/cache/l10n cannot delete non-empty directory: php-1.32.0-wmf.6/cache/l10n cannot delete non-empty directory: php-1.32.0-wmf.6/cache cannot delete non-empty directory: php-1.32.0-wmf.6/cache cannot delete non-empty directory: php-1.32.0-wmf.6 13:32:15 Finished rsync common (duration: 01m 55s) 13:32:15 Started cache_git_info 13:32:40 Finished cache_git_info (duration: 00m 24s) 13:32:40 Started sync-masters sync-masters: 100% (ok: 1; fail: 0; left: 0) 13:32:50 Finished sync-masters (duration: 00m 09s) 13:32:50 Started sync-pull-masters sync-pull-masters: 100% (ok: 1; fail: 0; left: 0) 13:32:58 Finished sync-pull-masters (duration: 00m 07s) 13:32:58 Started sync-check-canaries check-canaries: 100% (ok: 11; fail: 0; left: 0) 13:33:11 Finished Canaries Synced (duration: 00m 13s) 13:33:11 Executing check 'Check endpoints for mw1279.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mw1276.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mw1261.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mw1264.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mwdebug1002.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mwdebug1001.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mw1263.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mw1262.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mw1278.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mw1277.eqiad.wmnet' 13:33:12 Executing check 'Check endpoints for mw1265.eqiad.wmnet' 13:33:17 Finished Canary Endpoint Check Complete (duration: 00m 05s) 13:33:17 Waiting for canary traffic... 13:33:32 Executing check 'Logstash Error rate for mw1279.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mw1276.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mw1261.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mw1264.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mwdebug1002.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mwdebug1001.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mw1263.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mw1262.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mw1278.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mw1277.eqiad.wmnet' 13:33:32 Executing check 'Logstash Error rate for mw1265.eqiad.wmnet' 13:33:32 Finished sync-check-canaries (duration: 00m 34s) 13:33:32 Started sync-proxies sync-proxies: 100% (ok: 9; fail: 0; left: 0) 13:33:44 Finished sync-proxies (duration: 00m 11s) 13:33:44 Started sync-apaches 13:35:09 ['/usr/bin/scap', 'pull', '--no-update-l10n', 'mw1268.eqiad.wmnet', 'mw1314.eqiad.wmnet', 'mw2255.codfw.wmnet', 'mw2290.codfw.wmnet', 'mw2216.codfw.wmnet', 'mw1251.eqiad.wmnet', 'mw2188.codfw.wmnet', 'mw1320.eqiad.wmnet', 'mw1285.eqiad.wmnet'] on mwdebug2002.codfw.wmnet returned [70]: 13:34:03 Copying to mwdebug2002.codfw.wmnet from mw2255.codfw.wmnet 13:34:03 Started rsync common rsync: write failed on "/srv/mediawiki/wikiversions.json": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.2] rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/WikimediaMessages/i18n/wikimediainterwikisearchresults" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/WikimediaMessages/i18n/wikimediaoverrides" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/WikimediaMessages/i18n/wikimediaoverridesnotranslate" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/WikimediaMessages/i18n/wikimediaprojectnames" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/WikimediaMessages/i18n/wikitech" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/WikimediaMessages/modules" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/WikimediaMessages/tests" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/XAnalytics" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/ZeroBanner" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/ZeroPortal" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/cldr" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/intersection" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/timeline" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/extensions/wikihiero" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/images" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/includes" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/languages" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/maintenance" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/mw-config" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/resources" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/serialized" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/skins" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/tests" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir "/srv/mediawiki/php-1.32.0-wmf.13/vendor" failed: No space left on device (28) *** Skipping any contents from this failed directory *** 13:35:09 Finished rsync common (duration: 01m 06s) rsync error: error in file IO (code 11) at io.c(1642) [sender=3.1.2] 13:35:09 Unhandled error: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 339, in run exit_status = app.main(app.extra_arguments) File "/usr/lib/python2.7/dist-packages/scap/main.py", line 660, in main rsync_args=rsync_args File "/usr/lib/python2.7/dist-packages/scap/utils.py", line 402, in context_wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/scap/tasks.py", line 401, in sync_common subprocess.check_call(rsync) File "/usr/lib/python2.7/subprocess.py", line 186, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['sudo', '-u', 'mwdeploy', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/cache/l10n/*.cdb', '--exclude=*.swp', '--no-perms', '--exclude=**/.git', 'mw2255.codfw.wmnet::common', '/srv/mediawiki']' returned non-zero exit status 11 13:35:09 pull failed: Command '['sudo', '-u', 'mwdeploy', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/cache/l10n/*.cdb', '--exclude=*.swp', '--no-perms', '--exclude=**/.git', 'mw2255.codfw.wmnet::common', '/srv/mediawiki']' returned non-zero exit status 11 sync-apaches: 100% (ok: 262; fail: 1; left: 0) 13:35:09 1 apaches had sync errors 13:35:09 Finished sync-apaches (duration: 01m 25s) 13:35:09 Started clean-remote-caches clean-remote-caches: 100% (ok: 285; fail: 0; left: 0) 13:35:13 Finished clean-remote-caches (duration: 00m 04s) 13:35:13 Pruned MediaWiki: 1.32.0-wmf.6 (duration: 06m 18s)