Page MenuHomePhabricator

beta cluster scap failure
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50109/console

21:03:58 21:03:58 Started scap: beta-scap-eqiad (build #50109)
21:03:58 21:03:58 Copying to deployment-bastion.eqiad.wmflabs from deployment-bastion.eqiad.wmflabs
21:03:58 21:03:58 Started rsync common
21:04:07 21:04:07 Finished rsync common (duration: 00m 08s)
21:04:07 21:04:07 Compiled /srv/mediawiki/wikiversions-labs.json to /srv/mediawiki/wikiversions-labs.cdb
21:04:07 21:04:07 Started mw-update-l10n
21:04:07 21:04:07 Updating ExtensionMessages-master.php
21:04:08 21:04:08 Finished mw-update-l10n (duration: 00m 00s)
21:04:08 21:04:08 Unhandled error:
21:04:08 Traceback (most recent call last):
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/cli.py", line 276, in run
21:04:08     exit_status = app.main(extra_args)
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/main.py", line 39, in main
21:04:08     self._before_cluster_sync()
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/main.py", line 218, in _before_cluster_sync
21:04:08     version, wikidb, self.verbose, self.config)
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/tasks.py", line 496, in update_localization_cache
21:04:08     logger)
21:04:08   File "/mnt/srv/deployment/scap/scap/scap/utils.py", line 308, in sudo_check_call
21:04:08     raise subprocess.CalledProcessError(proc.returncode, cmd)
21:04:08 CalledProcessError: Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="eowiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.VpRueQKWIR" ' returned non-zero exit status 1

Event Timeline

greg assigned this task to mmodell.
greg raised the priority of this task from to Unbreak Now!.
greg updated the task description. (Show Details)
greg added subscribers: greg, thcipriani, dduvall, Jdlrobson.

@mmodell @thcipriani ping (this looks unrelated to the flappiness of wmf labs as it started before all of that)

It seems scap strip the underlying command output. Running mwscript manually I got:

Fatal error: Uncaught exception 'LogicException' with message 'Missing stream uri, the stream can not be opened.' in /mnt/srv/mediawiki-staging/php-master/includes/debug/logger/monolog/LegacyHandler.php:113
Stack trace:
#0 /mnt/srv/mediawiki-staging/php-master/includes/debug/logger/monolog/LegacyHandler.php(197): MediaWiki\Logger\Monolog\LegacyHandler->openSink()
#1 /mnt/srv/mediawiki-staging/php-master/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php(37): MediaWiki\Logger\Monolog\LegacyHandler->write(Array)
#2 /mnt/srv/mediawiki-staging/php-master/vendor/monolog/monolog/src/Monolog/Logger.php(265): Monolog\Handler\AbstractProcessingHandler->handle(Array)
#3 /mnt/srv/mediawiki-staging/php-master/vendor/monolog/monolog/src/Monolog/Logger.php(473): Monolog\Logger->addRecord(200, '[163d8839] [no ...', Array)
#4 /mnt/srv/mediawiki-staging/php-master/includes/GlobalFunctions.php(1157): Monolog\Logger->info('[163d8839] [no ...', Array)
#5 /mnt/srv/mediawiki-staging/php-master/includes/exception/MWExcepti in /mnt/srv/mediawiki-staging/php-master/includes/debug/logger/monolog/LegacyHandler.php on line 113

Which might be related to the code being outdated or some recent logging config change. Looking at the database update job, only testwiki seems to be able to run! And /usr/local/bin/mwscript eval.php --wiki="testwiki" works.

Filled as T97138

Magically fixed when T97138 got fixed :)