Page MenuHomePhabricator

beta cluster scap failure
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50109/console

21:03:58 21:03:58 Started scap: beta-scap-eqiad (build #50109)
21:03:58 21:03:58 Copying to deployment-bastion.eqiad.wmflabs from deployment-bastion.eqiad.wmflabs
21:03:58 21:03:58 Started rsync common
21:04:07 21:04:07 Finished rsync common (duration: 00m 08s)
21:04:07 21:04:07 Compiled /srv/mediawiki/wikiversions-labs.json to /srv/mediawiki/wikiversions-labs.cdb
21:04:07 21:04:07 Started mw-update-l10n
21:04:07 21:04:07 Updating ExtensionMessages-master.php
21:04:08 21:04:08 Finished mw-update-l10n (duration: 00m 00s)
21:04:08 21:04:08 Unhandled error:
21:04:08 Traceback (most recent call last):
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/cli.py", line 276, in run
21:04:08     exit_status = app.main(extra_args)
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/main.py", line 39, in main
21:04:08     self._before_cluster_sync()
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/main.py", line 218, in _before_cluster_sync
21:04:08     version, wikidb, self.verbose, self.config)
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/tasks.py", line 496, in update_localization_cache
21:04:08     logger)
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/utils.py", line 308, in sudo_check_call
21:04:08     raise subprocess.CalledProcessError(proc.returncode, cmd)
21:04:08 CalledProcessError: Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="eowiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.VpRueQKWIR" ' returned non-zero exit status 1

Event Timeline

greg created this task.Apr 22 2015, 9:11 PM
greg assigned this task to mmodell.
greg raised the priority of this task from to Unbreak Now!.
greg updated the task description. (Show Details)
greg added subscribers: greg, thcipriani, dduvall, Jdlrobson.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 22 2015, 9:11 PM
Jdlrobson set Security to None.Apr 22 2015, 10:07 PM
Jdlrobson added a subscriber: JKatzWMF.

@mmodell @thcipriani ping (this looks unrelated to the flappiness of wmf labs as it started before all of that)

hashar added a subscriber: hashar.EditedApr 24 2015, 2:47 PM

It seems scap strip the underlying command output. Running mwscript manually I got:

Fatal error: Uncaught exception 'LogicException' with message 'Missing stream uri, the stream can not be opened.' in /mnt/srv/mediawiki-staging/php-master/includes/debug/logger/monolog/LegacyHandler.php:113
Stack trace:
#0 /mnt/srv/mediawiki-staging/php-master/includes/debug/logger/monolog/LegacyHandler.php(197): MediaWiki\Logger\Monolog\LegacyHandler->openSink()
#1 /mnt/srv/mediawiki-staging/php-master/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php(37): MediaWiki\Logger\Monolog\LegacyHandler->write(Array)
#2 /mnt/srv/mediawiki-staging/php-master/vendor/monolog/monolog/src/Monolog/Logger.php(265): Monolog\Handler\AbstractProcessingHandler->handle(Array)
#3 /mnt/srv/mediawiki-staging/php-master/vendor/monolog/monolog/src/Monolog/Logger.php(473): Monolog\Logger->addRecord(200, '[163d8839] [no ...', Array)
#4 /mnt/srv/mediawiki-staging/php-master/includes/GlobalFunctions.php(1157): Monolog\Logger->info('[163d8839] [no ...', Array)
#5 /mnt/srv/mediawiki-staging/php-master/includes/exception/MWExcepti in /mnt/srv/mediawiki-staging/php-master/includes/debug/logger/monolog/LegacyHandler.php on line 113

Which might be related to the code being outdated or some recent logging config change. Looking at the database update job, only testwiki seems to be able to run! And /usr/local/bin/mwscript eval.php --wiki="testwiki" works.

Filled as T97138

hashar closed this task as Resolved.Apr 24 2015, 7:56 PM

Magically fixed when T97138 got fixed :)

greg moved this task from To Triage to Done on the Deployments board.Apr 29 2015, 4:40 PM