Page MenuHomePhabricator
Create Task
Maniphest T343291

[betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php
Closed, ResolvedPublic
Actions

Description

Beta cluster wiki displays the following on log in:

[ZMmXnt14SQ@J7Ce8BSAbxgAAABQ] /w/index.php?returnto=Benutzer+Diskussion:*Unregistered+15329&returntoquery=uselang%3Den&title=Spezial:Anmelden RuntimeException: Failed to run getConfiguration.php: No MWMultiVersion instance initialized! MWScript.php wrapper not used?
Backtrace:

from /srv/mediawiki/php-master/includes/SiteConfiguration.php(608)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(115): SiteConfiguration->getConfig(string, array)
#1 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1725): MediaWiki\JobQueue\JobQueueGroupFactory::MediaWiki\JobQueue\{closure}(boolean, integer, array, NULL, array)
#2 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1555): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#3 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(117): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#4 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(150): MediaWiki\JobQueue\JobQueueGroupFactory->getCachedJobClasses(string)
#5 /srv/mediawiki/php-master/includes/user/UserGroupManagerFactory.php(122): MediaWiki\JobQueue\JobQueueGroupFactory->makeJobQueueGroup(string)
#6 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2747): MediaWiki\User\UserGroupManagerFactory->getUserGroupManager(string)
#7 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2575): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->localUserData(string)
#8 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2544): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->queryAttached()
#9 /srv/mediawiki/php-master/extensions/CentralAuth/includes/Hooks/Handlers/PasswordPoliciesHookHandler.php(57): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->getLocalGroups()
#10 /srv/mediawiki/php-master/includes/HookContainer/HookContainer.php(160): MediaWiki\Extension\CentralAuth\Hooks\Handlers\PasswordPoliciesHookHandler->onPasswordPoliciesForUser(User, array)
#11 /srv/mediawiki/php-master/includes/HookContainer/HookRunner.php(3065): MediaWiki\HookContainer\HookContainer->run(string, array)
#12 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(191): MediaWiki\HookContainer\HookRunner->onPasswordPoliciesForUser(User, array)
#13 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(81): UserPasswordPolicy->getPoliciesForUser(User)
#14 /srv/mediawiki/php-master/includes/user/User.php(1017): UserPasswordPolicy->checkUserPassword(User, string)
#15 /srv/mediawiki/php-master/includes/auth/AbstractPasswordPrimaryAuthenticationProvider.php(109): User->checkPasswordValidity(string)
#16 /srv/mediawiki/php-master/includes/auth/TemporaryPasswordPrimaryAuthenticationProvider.php(161): MediaWiki\Auth\AbstractPasswordPrimaryAuthenticationProvider->checkPasswordValidity(string, string)
#17 /srv/mediawiki/php-master/includes/auth/AuthManager.php(532): MediaWiki\Auth\TemporaryPasswordPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
#18 /srv/mediawiki/php-master/includes/auth/AuthManager.php(462): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#19 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(374): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#20 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(506): AuthManagerSpecialPage->performAuthenticationStep(string, array)
#21 [internal function]: AuthManagerSpecialPage->handleFormSubmit(array, CodexHTMLForm)
#22 /srv/mediawiki/php-master/includes/htmlform/HTMLForm.php(748): call_user_func(array, array, CodexHTMLForm)
#23 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(437): HTMLForm->trySubmit()
#24 /srv/mediawiki/php-master/includes/specialpage/LoginSignupSpecialPage.php(317): AuthManagerSpecialPage->trySubmit()
#25 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(701): LoginSignupSpecialPage->execute(NULL)
#26 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(1564): SpecialPage->run(NULL)
#27 /srv/mediawiki/php-master/includes/MediaWiki.php(344): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#28 /srv/mediawiki/php-master/includes/MediaWiki.php(948): MediaWiki->performRequest()
#29 /srv/mediawiki/php-master/includes/MediaWiki.php(597): MediaWiki->main()
#30 /srv/mediawiki/php-master/index.php(50): MediaWiki->run()
#31 /srv/mediawiki/php-master/index.php(46): wfIndexMain()
#32 /srv/mediawiki/w/index.php(3): require(string)
#33 {main}

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClassesmediawiki/corewmf/1.41.0-wmf.22+46 -61
jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClassesmediawiki/coremaster+46 -61
shell: Do not sandbox MediaWiki maintenance scriptsmediawiki/coremaster+3 -1
shell: Do not use a shell in firejailmediawiki/coremaster+2 -0
shell: Always wrap maintenance scripts in mwscriptoperations/mediawiki-configmaster+1 -1
shell: Add $wgShellMaintenanceRunner optionmediawiki/coremaster+66 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Etonkovidova created this task.Aug 1 2023, 11:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 1 2023, 11:36 PM
Etonkovidova updated the task description. (Show Details)Aug 1 2023, 11:39 PM
hueitan subscribed.Aug 2 2023, 9:03 AM
TheresNoTime subscribed.Aug 2 2023, 9:19 AM
TheresNoTime added a comment.EditedAug 2 2023, 9:22 AM

Little suspicious of rMW420f22226656: Make Shell::makeScriptCommand use run.php (935734)

TheresNoTime added a comment.Aug 2 2023, 9:25 AM
Error
normalized_message
[{reqId}] {exception_url}   RuntimeException: Failed to run getConfiguration.php: No MWMultiVersion instance initialized! MWScript.php wrapper not used?
exception.trace
from /srv/mediawiki/php-master/includes/SiteConfiguration.php(608)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(115): SiteConfiguration->getConfig(string, array)
#1 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1725): MediaWiki\JobQueue\JobQueueGroupFactory::MediaWiki\JobQueue\{closure}(boolean, integer, array, NULL, array)
#2 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1555): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#3 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(117): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#4 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(150): MediaWiki\JobQueue\JobQueueGroupFactory->getCachedJobClasses(string)
#5 /srv/mediawiki/php-master/includes/user/UserGroupManagerFactory.php(122): MediaWiki\JobQueue\JobQueueGroupFactory->makeJobQueueGroup(string)
#6 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2747): MediaWiki\User\UserGroupManagerFactory->getUserGroupManager(string)
#7 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2575): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->localUserData(string)
#8 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2544): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->queryAttached()
#9 /srv/mediawiki/wmf-config/CommonSettings.php(2041): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->getLocalGroups()
#10 /srv/mediawiki/wmf-config/CommonSettings-labs.php(61): wmfGetPrivilegedGroups(User)
#11 /srv/mediawiki/php-master/includes/HookContainer/HookContainer.php(160): {closure}(User, array)
#12 /srv/mediawiki/php-master/includes/HookContainer/HookRunner.php(3065): MediaWiki\HookContainer\HookContainer->run(string, array)
#13 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(191): MediaWiki\HookContainer\HookRunner->onPasswordPoliciesForUser(User, array)
#14 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(81): UserPasswordPolicy->getPoliciesForUser(User)
#15 /srv/mediawiki/php-master/includes/user/User.php(1017): UserPasswordPolicy->checkUserPassword(User, string)
#16 /srv/mediawiki/php-master/includes/auth/AbstractPasswordPrimaryAuthenticationProvider.php(109): User->checkPasswordValidity(string)
#17 /srv/mediawiki/php-master/includes/auth/TemporaryPasswordPrimaryAuthenticationProvider.php(161): MediaWiki\Auth\AbstractPasswordPrimaryAuthenticationProvider->checkPasswordValidity(string, string)
#18 /srv/mediawiki/php-master/includes/auth/AuthManager.php(532): MediaWiki\Auth\TemporaryPasswordPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
#19 /srv/mediawiki/php-master/includes/auth/AuthManager.php(462): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#20 /srv/mediawiki/php-master/includes/api/ApiClientLogin.php(103): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#21 /srv/mediawiki/php-master/includes/api/ApiMain.php(1921): ApiClientLogin->execute()
#22 /srv/mediawiki/php-master/includes/api/ApiMain.php(898): ApiMain->executeAction()
#23 /srv/mediawiki/php-master/includes/api/ApiMain.php(869): ApiMain->executeActionWithErrorHandling()
#24 /srv/mediawiki/php-master/api.php(95): ApiMain->execute()
#25 /srv/mediawiki/php-master/api.php(48): wfApiMain()
#26 /srv/mediawiki/w/api.php(3): require(string)
#27 {main}
Impact
Notes
Ladsgroup subscribed.Aug 2 2023, 9:34 AM
Urbanecm_WMF subscribed.Aug 2 2023, 9:50 AM
Quiddity subscribed.Aug 2 2023, 4:09 PM
vaughnwalters subscribed.EditedAug 2 2023, 7:50 PM

This also applies to temp users. When attempting to edit an article on https://de.wikipedia.beta.wmflabs.org/ as an anon user where it should create a temp user for IP masking, I get the same message:

[ZMqyjappClPgoVLMgOV8@gAAAA0] /w/index.php?action=submit&title=%F0%9F%87%AC%F0%9F%87%B1 RuntimeException: Failed to run getConfiguration.php: No MWMultiVersion instance initialized! MWScript.php wrapper not used?
Backtrace:

from /srv/mediawiki/php-master/includes/SiteConfiguration.php(608)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(115): SiteConfiguration->getConfig(string, array)
#1 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1725): MediaWiki\JobQueue\JobQueueGroupFactory::MediaWiki\JobQueue\{closure}(boolean, integer, array, NULL, array)
#2 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1555): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#3 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(117): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#4 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(150): MediaWiki\JobQueue\JobQueueGroupFactory->getCachedJobClasses(string)
#5 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUtilityService.php(169): MediaWiki\JobQueue\JobQueueGroupFactory->makeJobQueueGroup(string)
#6 /srv/mediawiki/php-master/extensions/CentralAuth/includes/Hooks/Handlers/UserCreationHookHandler.php(75): MediaWiki\Extension\CentralAuth\CentralAuthUtilityService->scheduleCreationJobs(MediaWiki\Extension\CentralAuth\User\CentralAuthUser)
#7 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(2183): MediaWiki\Extension\CentralAuth\Hooks\Handlers\UserCreationHookHandler->MediaWiki\Extension\CentralAuth\Hooks\Handlers\{closure}(integer, Wikimedia\Rdbms\DatabaseMysqli)
#8 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1574): Wikimedia\Rdbms\Database->runOnTransactionIdleCallbacks(integer, array)
#9 /srv/mediawiki/php-master/includes/libs/rdbms/lbfactory/LBFactory.php(382): Wikimedia\Rdbms\LoadBalancer->runPrimaryTransactionIdleCallbacks(string)
#10 /srv/mediawiki/php-master/includes/libs/rdbms/lbfactory/LBFactory.php(339): Wikimedia\Rdbms\LBFactory->executePostTransactionCallbacks()
#11 /srv/mediawiki/php-master/includes/MediaWiki.php(708): Wikimedia\Rdbms\LBFactory->commitPrimaryChanges(string, integer)
#12 /srv/mediawiki/php-master/includes/MediaWiki.php(678): MediaWiki::preOutputCommit(RequestContext)
#13 /srv/mediawiki/php-master/includes/MediaWiki.php(961): MediaWiki->doPreOutputCommit()
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(597): MediaWiki->main()
#15 /srv/mediawiki/php-master/index.php(50): MediaWiki->run()
#16 /srv/mediawiki/php-master/index.php(46): wfIndexMain()
#17 /srv/mediawiki/w/index.php(3): require(string)
#18 {main}
TheresNoTime added a comment.EditedAug 2 2023, 8:30 PM
normalized_message
Executing: /bin/bash '/srv/mediawiki/php-master/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/bin/firejail'\'' '\''--quiet'\'' '\''--profile=/srv/mediawiki/php-master/includes/shell/firejail.profile'\'' '\''--noroot'\'' '\''--seccomp'\'' '\''--private-dev'\'' -- '\''/usr/bin/php'\'' '\''/srv/mediawiki/php-master/maintenance/run.php'\'' '\''getConfiguration'\'' '\''--wiki'\'' '\''arwiki'\'' '\''--settings'\'' '\''wgJobClasses'\'' '\''--format'\'' '\''PHP'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=50; SB_CGROUP='\''/sys/fs/cgroup/memory/mediawiki/job'\''; SB_MEM_LIMIT=0; SB_FILE_SIZE_LIMIT=0; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes'

Running

samtar@deployment-mwmaint02:~$ php /srv/mediawiki/php-master/maintenance/run.php getConfiguration --wiki arwiki

gives the reported error of No MWMultiVersion instance initialized! MWScript.php wrapper not used?

In comparison, running

samtar@deployment-mwmaint02:~$ mwscript getConfiguration --wiki arwiki --format json

returns as expected ({"wg3dProcessEnviron":{ ...)

rMWab802a66e685: jobqueue: Use a proper config system for dealing with settings (p.1) introduced getCachedJobClasses (with a note that "SiteConfiguration will have to be refactored to use our config system")

TheresNoTime added a comment.Aug 2 2023, 8:51 PM

Revert of rMWab802a66e685: jobqueue: Use a proper config system for dealing with settings (p.1) submitted at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/944855

TheresNoTime added a comment.Aug 2 2023, 9:10 PM

Temporarily doing

diff --git a/includes/jobqueue/JobQueueGroupFactory.php b/includes/jobqueue/JobQueueGroupFactory.php
index 680c1638782..cb8df364503 100644
--- a/includes/jobqueue/JobQueueGroupFactory.php
+++ b/includes/jobqueue/JobQueueGroupFactory.php
@@ -147,7 +147,7 @@ class JobQueueGroupFactory {
                        $this->instances[$domain] = new JobQueueGroup(
                                $domain,
                                $this->readOnlyMode,
-                               $this->getCachedJobClasses( $domain ),
+                               $this->options->get( MainConfigNames::JobClasses ),
                                $this->options->get( MainConfigNames::JobTypeConf ),
                                $this->options->get( MainConfigNames::JobTypesExcludedFromDefaultQueue ),
                                $this->statsdDataFactory,

resolved the issue, so its getCachedJobClasses() (or more accurately, $wgConf->getConfig() not handling multi-wiki stuff well)

TheresNoTime triaged this task as Unbreak Now! priority.Aug 2 2023, 9:13 PM
TheresNoTime added a parent task: T340249: 1.41.0-wmf.21 deployment blockers.

I could be wrong, but as it stands without that revert, this'll break things when we roll to .21?

phuedx mentioned this in T343372: Selenium daily build tests failing with Node.js error.Aug 3 2023, 4:59 AM
Michael subscribed.Aug 3 2023, 8:50 AM
Michael mentioned this in T340832: [STORY] MUL - While reading, make Wikibase label fallbacks or the `mul` language code default clearly visible.Aug 3 2023, 8:53 AM
Jdlrobson subscribed.Aug 3 2023, 2:45 PM
Tgr subscribed.Aug 3 2023, 6:32 PM

I feel if $wgConf->getConfig() would be broken in production, we'd have noticed already.

Tgr added a comment.Aug 3 2023, 6:34 PM

Actually JobQueueGroupFactory::getCachedConfigVar() seems to be the only thing in existence calling it. Huh.

• Jrbranaa subscribed.Aug 3 2023, 8:54 PM
Samwalton9-WMF subscribed.Aug 4 2023, 12:49 PM
Novem_Linguae merged a task: T343532: (Beta Cluster) RuntimeException: Failed to run getConfiguration.php: No MWMultiVersion instance initialized.Aug 4 2023, 12:56 PM
Novem_Linguae subscribed.
gerritbot added a comment.Aug 4 2023, 11:22 PM

Change 945878 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] shell: Add $wgShellMaintenanceRunner option

https://gerrit.wikimedia.org/r/945878

gerritbot added a project: Patch-For-Review.Aug 4 2023, 11:22 PM
gerritbot added a comment.Aug 4 2023, 11:26 PM

Change 945878 abandoned by Gergő Tisza:

[mediawiki/core@master] shell: Add $wgShellMaintenanceRunner option

Reason:

https://gerrit.wikimedia.org/r/945878

Maintenance_bot removed a project: Patch-For-Review.Aug 4 2023, 11:30 PM
gerritbot added a comment.Aug 4 2023, 11:43 PM

Change 945883 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] shell: Always wrap maintenance scripts in mwscript

https://gerrit.wikimedia.org/r/945883

gerritbot added a project: Patch-For-Review.Aug 4 2023, 11:43 PM
XenonX3 subscribed.Aug 5 2023, 9:12 PM
Krinkle subscribed.Aug 7 2023, 3:01 AM
In T343291#9064640, @TheresNoTime wrote:

Temporarily doing

diff --git a/includes/jobqueue/JobQueueGroupFactory.php b/includes/jobqueue/JobQueueGroupFactory.php
index 680c1638782..cb8df364503 100644
--- a/includes/jobqueue/JobQueueGroupFactory.php
+++ b/includes/jobqueue/JobQueueGroupFactory.php
@@ -147,7 +147,7 @@ class JobQueueGroupFactory {
                        $this->instances[$domain] = new JobQueueGroup(
                                $domain,
                                $this->readOnlyMode,
-                               $this->getCachedJobClasses( $domain ),
+                               $this->options->get( MainConfigNames::JobClasses ),
                                $this->options->get( MainConfigNames::JobTypeConf ),
                                $this->options->get( MainConfigNames::JobTypesExcludedFromDefaultQueue ),
                                $this->statsdDataFactory,

resolved the issue, so its getCachedJobClasses() (or more accurately, $wgConf->getConfig() not handling multi-wiki stuff well)

You might know this already, but given that the proposed revert at change 944855 is still open, I'll write down a potential reason that this revert won't fix the issue, and that the above diff could cause a bug in production.

Both before and after change 937045, $wgConf->getConfig() is used to retreive $wgJobClasses for foreign wikis.

Before this change, as the revert restores, the value for the "current" wiki is injected from JobQueueGroupFactory to the JobQueueGroup parameter, which then subsequently ignores it for most wikis, and calls $wgConfig->getConfig() instead. After this change, JobQueueGroupFactory calls $wgConfig->getConfig() as-needed, and injects the correct value for JobQueueGroup to then safely use.

The above diff forces the local wiki value of wgJobClasses unto other wiki contexts, thus avoiding the wgConf call, but the full revert would not have avoided that call, I think.

gerritbot added a comment.Aug 7 2023, 7:59 AM

Change 945883 merged by jenkins-bot:

[operations/mediawiki-config@master] shell: Always wrap maintenance scripts in mwscript

https://gerrit.wikimedia.org/r/945883

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2023, 8:10 AM
TheresNoTime added a comment.Aug 7 2023, 11:22 AM
In T343291#9072561, @Krinkle wrote:
In T343291#9064640, @TheresNoTime wrote:

Temporarily doing

diff --git a/includes/jobqueue/JobQueueGroupFactory.php b/includes/jobqueue/JobQueueGroupFactory.php
index 680c1638782..cb8df364503 100644
--- a/includes/jobqueue/JobQueueGroupFactory.php
+++ b/includes/jobqueue/JobQueueGroupFactory.php
@@ -147,7 +147,7 @@ class JobQueueGroupFactory {
                        $this->instances[$domain] = new JobQueueGroup(
                                $domain,
                                $this->readOnlyMode,
-                               $this->getCachedJobClasses( $domain ),
+                               $this->options->get( MainConfigNames::JobClasses ),
                                $this->options->get( MainConfigNames::JobTypeConf ),
                                $this->options->get( MainConfigNames::JobTypesExcludedFromDefaultQueue ),
                                $this->statsdDataFactory,

resolved the issue, so its getCachedJobClasses() (or more accurately, $wgConf->getConfig() not handling multi-wiki stuff well)

You might know this already, but given that the proposed revert at change 944855 is still open, I'll write down a potential reason that this revert won't fix the issue, and that the above diff could cause a bug in production.

Both before and after change 937045, $wgConf->getConfig() is used to retreive $wgJobClasses for foreign wikis.

Before this change, as the revert restores, the value for the "current" wiki is injected from JobQueueGroupFactory to the JobQueueGroup parameter, which then subsequently ignores it for most wikis, and calls $wgConfig->getConfig() instead. After this change, JobQueueGroupFactory calls $wgConfig->getConfig() as-needed, and injects the correct value for JobQueueGroup to then safely use.

The above diff forces the local wiki value of wgJobClasses unto other wiki contexts, thus avoiding the wgConf call, but the full revert would not have avoided that call, I think.

I'll defer to your knowledge on this @Krinkle — either way, it'd be nice if we fixed this.. r/945883 above didn't resolve the issue

zeljkofilipin added a project: User-zeljkofilipin.Aug 7 2023, 12:14 PM
zeljkofilipin moved this task from Backlog 🪒 to Watching 📺 on the User-zeljkofilipin board.
zeljkofilipin subscribed.
Tgr added a comment.EditedAug 7 2023, 5:01 PM

It seems to have fixed the specific issue and triggered a new one:

tgr@deployment-mwmaint02:~$ mwscript shell.php aawiki

> $command = \MediaWiki\Shell\Shell::makeScriptCommand( 'getConfiguration', [ '--wiki', 'enwiki', '--settings', 'wgJobClasses', '--format', 'PHP' ] )

> (string)$command
= "#Command: '/usr/bin/php' '/srv/mediawiki/multiversion/MWScript.php' 'getConfiguration' '--wiki' 'enwiki' '--settings' 'wgJobClasses' '--format' 'PHP'"

> $result = $command->execute()
> $result->getStderr()
= "execvp: Permission denied\n"
Tgr added a comment.Aug 7 2023, 5:05 PM

This works just fine:

tgr@deployment-mwmaint02:~$ sudo -u www-data '/usr/bin/php' '/srv/mediawiki/multiversion/MWScript.php' 'getConfiguration' '--wiki' 'enwiki' '--settings' 'wgJobClasses' '--format' 'PHP'

Wrong user, or wrong firejail configuration, or something like that? Might or might not be an issue in production.

Tgr added a comment.Aug 7 2023, 5:23 PM

This works:

> $command = \MediaWiki\Shell\Shell::makeScriptCommand( 'getConfiguration', [ '--wiki', 'enwiki', '--settings', 'wgJobClasses', '--format', 'PHP' ] )
> $command->disableSandbox()->execute()->getStdOut()

This doesn't work:

> $command = \MediaWiki\Shell\Shell::makeScriptCommand( 'getConfiguration', [ '--wiki', 'enwiki', '--settings', 'wgJobClasses', '--format', 'PHP' ] )
> $command->restrict( 0 )->disableSandbox( false )->execute()->getStdOut()

so definitely Firejail-related.

The issue does affect production (I tested on mwmaint, not an appserver, but I doubt that would make a difference).

Tgr added a comment.Aug 7 2023, 5:28 PM

Unlike the previous error, this doesn't seem to be related to rMW420f22226656: Make Shell::makeScriptCommand use run.php btw:

> $command = \MediaWiki\Shell\Shell::makeScriptCommand( MW_INSTALL_PATH . '/maintenance/getConfiguration.php', [ '--wiki', 'enwiki', '--settings', 'wgJobClasses', '--format', 'PHP' ] )
> $command->execute()->getStderr()
= "execvp: Permission denied\n"
In T343291#9072561, @Krinkle wrote:

Both before and after change 937045, $wgConf->getConfig() is used to retreive $wgJobClasses for foreign wikis.

I guess the upshot is that this never actually worked and we just didn't notice? Unless something recently changed with Firejail or OS permissions configuration.

Tgr added a parent task: T343724: 1.41.0-wmf.22 deployment blockers.Aug 7 2023, 5:30 PM
Tgr added a comment.Aug 7 2023, 6:47 PM

The actual command that gets run is

/bin/bash '/srv/mediawiki/php-master/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/bin/firejail'\'' '\''--quiet'\'' '\''--profile=/srv/mediawiki/php-master/includes/shell/firejail.profile'\'' '\''--noroot'\'' '\''--seccomp'\'' '\''--private-dev'\'' -- '\''/usr/bin/php'\'' '\''/srv/mediawiki/multiversion/MWScript.php'\'' '\''getConfiguration'\'' '\''--wiki'\'' '\''enwiki'\'' '\''--settings'\'' '\''wgJobClasses'\'' '\''--format'\'' '\''PHP'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=50; SB_CGROUP='\''/sys/fs/cgroup/memory/mediawiki/job'\''; SB_MEM_LIMIT=1073741824; SB_FILE_SIZE_LIMIT=536870912; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes'⏎

A more minimal reproduction step is

sudo -u www-data /usr/bin/firejail --quiet --profile=/srv/mediawiki/php-master/includes/shell/firejail.profile -- /usr/bin/php /srv/mediawiki/w/health-check.php

In debug mode it says

Running '/usr/bin/php' '/srv/mediawiki/w/health-check.php'  command through /usr/sbin/nologin
execvp argument 0: /usr/sbin/nologin
execvp argument 1: -c
execvp argument 2: --
execvp argument 3: '/usr/bin/php' '/srv/mediawiki/w/health-check.php' 
Child process initialized in 71.05 ms
execvp: Permission denied

And indeed adding --shell=/bin/sh or --shell=none to the firejail command makes the script work.

gerritbot added a comment.Aug 7 2023, 6:57 PM

Change 946607 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] shell: Do not use a shell in firejail

https://gerrit.wikimedia.org/r/946607

gerritbot added a project: Patch-For-Review.Aug 7 2023, 6:57 PM
Tgr added a comment.Aug 7 2023, 7:12 PM

Testing with the patch manually applied, I get

Warning: require(/srv/mediawiki/wmf-config/../private/PrivateSettings.php): failed to open stream: Permission denied in /srv/mediawiki/wmf-config/CommonSettings.php on line 319
Fatal error: require(): Failed opening required '/srv/mediawiki/wmf-config/../private/PrivateSettings.php' (include_path='/srv/mediawiki/php-master/vendor/pear/console_getopt:/srv/mediawiki/php-master/vendor/pear/mail:/srv/mediawiki/php-master/vendor/pear/mail_mime:/srv/mediawiki/php-master/vendor/pear/net_smtp:/srv/mediawiki/php-master/vendor/pear/net_socket:/srv/mediawiki/php-master/vendor/pear/net_url2:/srv/mediawiki/php-master/vendor/pear/pear-core-minimal/src:/srv/mediawiki/php-master/vendor/pear/pear_exception:/srv/mediawiki/php-master:/usr/local/lib/php:/usr/share/php') in /srv/mediawiki/wmf-config/CommonSettings.php on line 319

...which I think makes that a dead-end. We explicitly ban /srv/mediawiki/private access in /etc/firejail/mediawiki.local and it doesn't seem like a good idea to change that.

Arguably there isn't really any point to use firejail for running a MediaWiki maintenance script - if MediaWiki is compromised it can just circumvent firejail in the first place. So probably we should just not use it in this case.

gerritbot added a comment.Aug 7 2023, 7:13 PM

Change 946607 abandoned by Gergő Tisza:

[mediawiki/core@master] shell: Do not use a shell in firejail

Reason:

on second thought probably not the right way to fix T343291

https://gerrit.wikimedia.org/r/946607

gerritbot added a comment.Aug 7 2023, 7:19 PM

Change 946610 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] shell: Do not sandbox MediaWiki maintenance scripts

https://gerrit.wikimedia.org/r/946610

RhinosF1 merged a task: T343482: can not create account at wikifunctions.beta.wmflabs.org.Aug 8 2023, 6:41 AM
RhinosF1 added subscribers: Bugreporter, RhinosF1, 0xDeadbeef.
gerritbot added a comment.Aug 8 2023, 10:55 AM

Change 946610 merged by jenkins-bot:

[mediawiki/core@master] shell: Do not sandbox MediaWiki maintenance scripts

https://gerrit.wikimedia.org/r/946610

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.22; 2023-08-15).Aug 8 2023, 11:00 AM
tstarling closed this task as Resolved.Aug 8 2023, 11:02 AM
tstarling claimed this task.
Maintenance_bot removed a project: Patch-For-Review.Aug 8 2023, 11:10 AM
Krinkle mentioned this in T344223: User logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found".Aug 15 2023, 11:47 AM
gerritbot added a comment.Aug 15 2023, 8:01 PM

Change 949103 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949103

gerritbot added a project: Patch-For-Review.Aug 15 2023, 8:01 PM
gerritbot added a comment.Aug 16 2023, 8:42 AM

Change 949178 had a related patch set uploaded (by D3r1ck01; author: Krinkle):

[mediawiki/core@wmf/1.41.0-wmf.22] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949178

gerritbot added a comment.Aug 16 2023, 8:52 AM

Change 949103 merged by jenkins-bot:

[mediawiki/core@master] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949103

gerritbot added a comment.Aug 16 2023, 1:43 PM

Change 949178 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.22] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949178

Stashbot mentioned this in T344143: New Impact module: Run backend updating logic on all Wikipedias.Aug 16 2023, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:52:19Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]]

Stashbot added a comment.Aug 16 2023, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:53:55Z] <urbanecm@deploy1002> urbanecm and d3r1ck01: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessibl

Stashbot added a comment.Aug 16 2023, 2:06 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-16T14:06:32Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] (duration: 14m 13s)

Maintenance_bot removed a project: Patch-For-Review.Aug 16 2023, 2:10 PM
zeljkofilipin removed a project: User-zeljkofilipin.Aug 18 2023, 12:19 PM
zeljkofilipin unsubscribed.
ReleaseTaggerBot edited projects, added MW-1.41-notes (1.41.0-wmf.25; 2023-09-05); removed MW-1.41-notes (1.41.0-wmf.22; 2023-08-15).Aug 29 2023, 6:10 PM
DAlangi_WMF mentioned this in T289543: Remove SiteConfiguration::getConfig (Fetch config via shell command).Sep 1 2023, 1:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits