Page MenuHomePhabricator
Create Task
Maniphest T379628

MW script "eval.php" failing for "testcommonswiki" during train operations
Closed, ResolvedPublic
Actions

Description

Train presync currently fails consistently with:

13:43:57 Checking for new runtime errors locally
'mwscript eval.php --wiki testcommonswiki' generated unexpected output: [09ce253dddc525fa8d86a8b6] [no req]   UnexpectedValueException: Section 's4' has no hosts defined.
Backtrace:
from /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(177)
#0 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(226): Wikimedia\Rdbms\LBFactoryMulti->newMainLB(bool)
#1 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/db/MWLBFactory.php(420): Wikimedia\Rdbms\LBFactoryMulti->getMainLB()
#2 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/ServiceWiring.php(731): MWLBFactory->applyGlobalState(Wikimedia\Rdbms\LBFactoryMulti, MediaWiki\Config\GlobalVarConfig, Wikimedia\Stats\BufferingStatsdDataFactory)
#3 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(442): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#4 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(406): Wikimedia\Services\ServiceContainer->createService(string)
#5 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService(string)
#6 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(1064): MediaWiki\MediaWikiServices->getService(string)
#7 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/ServiceWiring.php(1407): MediaWiki\MediaWikiServices->getDBLoadBalancerFactory()
#8 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(442): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#9 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(406): Wikimedia\Services\ServiceContainer->createService(string)
#10 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService(string)
#11 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(1495): MediaWiki\MediaWikiServices->getService(string)
#12 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/ServiceWiring.php(501): MediaWiki\MediaWikiServices->getNameTableStoreFactory()
#13 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(442): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#14 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(406): Wikimedia\Services\ServiceContainer->createService(string)
#15 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService(string)
#16 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(414): MediaWiki\MediaWikiServices->getService(string)
#17 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/object-factory/src/ObjectFactory.php(204): Wikimedia\Services\ServiceContainer->get(string)
#18 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/object-factory/src/ObjectFactory.php(149): Wikimedia\ObjectFactory\ObjectFactory::getObjectFromSpec(array, array)
#19 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookContainer.php(256): Wikimedia\ObjectFactory\ObjectFactory->createObject(array)
#20 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookContainer.php(351): MediaWiki\HookContainer\HookContainer->makeExtensionHandlerCallback(string, array, array)
#21 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookContainer.php(527): MediaWiki\HookContainer\HookContainer->normalizeHandler(string, array, array)
#22 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookContainer.php(146): MediaWiki\HookContainer\HookContainer->getHandlers(string, array)
#23 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookRunner.php(3496): MediaWiki\HookContainer\HookContainer->run(string, array)
#24 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/Setup.php(454): MediaWiki\HookContainer\HookRunner->onSetupAfterCache()
#25 /srv/mediawiki-staging/php-1.44.0-wmf.3/maintenance/run.php(49): require_once(string)
#26 /srv/mediawiki-staging/multiversion/MWScript.php(158): require_once(string)
#27 {main}

This is the same deployment stage where error T379044 was happening last week for "testwiki". It seems there is no connection and it's most likely just a coincidence. Do note however the MW script mentioned above is running inside a Docker container.

Interestingly enough, there were a couple of backports this morning and they were not affected by this.

Related Objects
Search...

Event Timeline

jnuche created this task.Nov 12 2024, 2:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 12 2024, 2:01 PM
jnuche triaged this task as Unbreak Now! priority.Nov 12 2024, 2:01 PM
jnuche added a parent task: T375662: 1.44.0-wmf.3 deployment blockers.
Urbanecm_WMF subscribed.Nov 12 2024, 2:04 PM

This also started to affect backports:

14:02:56 Started scap sync-world: Backport for [[gerrit:1090455|[CirrusSearch] testwiki: enable offloading weighted tags via EventBus (T378983)]]
14:02:56 Started cache_git_info
14:02:58 Finished cache_git_info (duration: 00m 01s)
14:02:58 Started l10n-update
14:02:59 Updating ExtensionMessages-1.44.0-wmf.2.php
14:03:05 Updating LocalisationCache for 1.44.0-wmf.2 using 30 thread(s)
14:03:05 Running rebuildLocalisationCache.php
14:03:07 0 languages rebuilt out of 524
14:03:07 Use --force to rebuild the caches which are still fresh.
14:03:08 Generating JSON versions and md5 files (as www-data)
14:03:11 Updating ExtensionMessages-1.44.0-wmf.3.php
14:03:16 Updating LocalisationCache for 1.44.0-wmf.3 using 30 thread(s)
14:03:16 Running rebuildLocalisationCache.php
14:03:18 0 languages rebuilt out of 524
14:03:18 Use --force to rebuild the caches which are still fresh.
14:03:19 Generating JSON versions and md5 files (as www-data)
14:03:20 Finished l10n-update (duration: 00m 22s)
14:03:20 Checking for new runtime errors locally
'mwscript eval.php --wiki testcommonswiki' generated unexpected output: [d5d7928621423095e090bd9e] [no req]   UnexpectedValueException: Section 's4' has no hosts defined.
Backtrace:
from /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(177)
#0 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(226): Wikimedia\Rdbms\LBFactoryMulti->newMainLB(bool)
#1 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/db/MWLBFactory.php(420): Wikimedia\Rdbms\LBFactoryMulti->getMainLB()
#2 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/ServiceWiring.php(731): MWLBFactory->applyGlobalState(Wikimedia\Rdbms\LBFactoryMulti, MediaWiki\Config\GlobalVarConfig, Wikimedia\Stats\BufferingStatsdDataFactory)
#3 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(442): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#4 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(406): Wikimedia\Services\ServiceContainer->createService(string)
#5 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService(string)
#6 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(1064): MediaWiki\MediaWikiServices->getService(string)
#7 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/ServiceWiring.php(1407): MediaWiki\MediaWikiServices->getDBLoadBalancerFactory()
#8 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(442): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#9 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(406): Wikimedia\Services\ServiceContainer->createService(string)
#10 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService(string)
#11 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(1495): MediaWiki\MediaWikiServices->getService(string)
#12 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/ServiceWiring.php(501): MediaWiki\MediaWikiServices->getNameTableStoreFactory()
#13 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(442): Wikimedia\Services\ServiceContainer::{closure}(MediaWiki\MediaWikiServices)
#14 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(406): Wikimedia\Services\ServiceContainer->createService(string)
#15 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService(string)
#16 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/services/src/ServiceContainer.php(414): MediaWiki\MediaWikiServices->getService(string)
#17 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/object-factory/src/ObjectFactory.php(204): Wikimedia\Services\ServiceContainer->get(string)
#18 /srv/mediawiki-staging/php-1.44.0-wmf.3/vendor/wikimedia/object-factory/src/ObjectFactory.php(149): Wikimedia\ObjectFactory\ObjectFactory::getObjectFromSpec(array, array)
#19 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookContainer.php(256): Wikimedia\ObjectFactory\ObjectFactory->createObject(array)
#20 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookContainer.php(351): MediaWiki\HookContainer\HookContainer->makeExtensionHandlerCallback(string, array, array)
#21 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookContainer.php(527): MediaWiki\HookContainer\HookContainer->normalizeHandler(string, array, array)
#22 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookContainer.php(146): MediaWiki\HookContainer\HookContainer->getHandlers(string, array)
#23 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/HookContainer/HookRunner.php(3496): MediaWiki\HookContainer\HookContainer->run(string, array)
#24 /srv/mediawiki-staging/php-1.44.0-wmf.3/includes/Setup.php(454): MediaWiki\HookContainer\HookRunner->onSetupAfterCache()
#25 /srv/mediawiki-staging/php-1.44.0-wmf.3/maintenance/run.php(49): require_once(string)
#26 /srv/mediawiki-staging/multiversion/MWScript.php(158): require_once(string)
#27 {main}

14:03:23 backport failed: <CalledProcessError> Command '['/usr/bin/scap', 'sync-world', 'Backport for [[gerrit:1090455|[CirrusSearch] testwiki: enable offloading weighted tags via EventBus (T378983)]]']' returned non-zero exit status 1. (scap version: 4.122.0)
[urbanecm@deploy2002 ~]$
jnuche added a comment.EditedNov 12 2024, 2:08 PM

Script works outside of the container apparently:

<tgr|away> Gergő Tisza running it by hand seems to work fine for me
3:05 PM eval.php I mean

EDIT: Or then again, maybe not:

<urbanecm> Martin Urbanec
tgr|away: running the full actual command indeed fails
<urbanecm> Martin Urbanec
jnuche: `sudo -u mwbuilder -n -- /usr/bin/scap mwscript --no-local-config --directory /srv/mediawiki-staging --user www-data --network -- eval.php --wiki=testcommonswiki

• Tgr subscribed.Nov 12 2024, 2:29 PM

See also T379589: scap backport fails at purgeMessageBlobStore.php with getaddrinfo failed which seems to have the same cause (using a mock DB config for offline operations) but occurred at a later scap step. And another scap attempt worked fine yesterday.

CDanis added subscribers: bvibber, CCiufo-WMF, Seddon and 2 others.Nov 12 2024, 2:49 PM
CDanis subscribed.

@bvibber @aude @Jdlrobson @CCiufo-WMF @Seddon FYI

brennen added projects: Release-Engineering-Team (Radar), User-brennen.Nov 12 2024, 4:07 PM
brennen subscribed.
Jdforrester-WMF subscribed.Nov 12 2024, 4:22 PM
dancy subscribed.Nov 12 2024, 4:37 PM

scap 4.123.0 has been deployed which should address this problem.

brennen closed this task as Resolved.Nov 12 2024, 5:27 PM
brennen claimed this task.

scap 4.123.0 has been deployed which should address this problem.

$ /usr/bin/scap stage-train -Dfull_image_build:True --yes auto
...
17:26:01 Finished scap sync-world: testwikis to 1.44.0-wmf.3  refs T375662 (duration: 45m 29s)
17:26:03 ==================================================
Checking version on https://test.wikipedia.org/wiki/Special:Version
Expected: 1.44.0-wmf.3
Actual:   1.44.0-wmf.3
Result:   SUCCESS

Looks good - thanks!

brennen reassigned this task from brennen to dancy.Nov 12 2024, 5:27 PM
thcipriani added a project: Essential-Work.May 8 2025, 1:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits