[betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Etonkovidova
	Aug 1 2023, 11:36 PM

Description

Beta cluster wiki displays the following on log in:

[ZMmXnt14SQ@J7Ce8BSAbxgAAABQ] /w/index.php?returnto=Benutzer+Diskussion:*Unregistered+15329&returntoquery=uselang%3Den&title=Spezial:Anmelden RuntimeException: Failed to run getConfiguration.php: No MWMultiVersion instance initialized! MWScript.php wrapper not used?
Backtrace:

from /srv/mediawiki/php-master/includes/SiteConfiguration.php(608)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(115): SiteConfiguration->getConfig(string, array)
#1 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1725): MediaWiki\JobQueue\JobQueueGroupFactory::MediaWiki\JobQueue\{closure}(boolean, integer, array, NULL, array)
#2 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1555): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#3 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(117): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#4 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(150): MediaWiki\JobQueue\JobQueueGroupFactory->getCachedJobClasses(string)
#5 /srv/mediawiki/php-master/includes/user/UserGroupManagerFactory.php(122): MediaWiki\JobQueue\JobQueueGroupFactory->makeJobQueueGroup(string)
#6 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2747): MediaWiki\User\UserGroupManagerFactory->getUserGroupManager(string)
#7 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2575): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->localUserData(string)
#8 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2544): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->queryAttached()
#9 /srv/mediawiki/php-master/extensions/CentralAuth/includes/Hooks/Handlers/PasswordPoliciesHookHandler.php(57): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->getLocalGroups()
#10 /srv/mediawiki/php-master/includes/HookContainer/HookContainer.php(160): MediaWiki\Extension\CentralAuth\Hooks\Handlers\PasswordPoliciesHookHandler->onPasswordPoliciesForUser(User, array)
#11 /srv/mediawiki/php-master/includes/HookContainer/HookRunner.php(3065): MediaWiki\HookContainer\HookContainer->run(string, array)
#12 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(191): MediaWiki\HookContainer\HookRunner->onPasswordPoliciesForUser(User, array)
#13 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(81): UserPasswordPolicy->getPoliciesForUser(User)
#14 /srv/mediawiki/php-master/includes/user/User.php(1017): UserPasswordPolicy->checkUserPassword(User, string)
#15 /srv/mediawiki/php-master/includes/auth/AbstractPasswordPrimaryAuthenticationProvider.php(109): User->checkPasswordValidity(string)
#16 /srv/mediawiki/php-master/includes/auth/TemporaryPasswordPrimaryAuthenticationProvider.php(161): MediaWiki\Auth\AbstractPasswordPrimaryAuthenticationProvider->checkPasswordValidity(string, string)
#17 /srv/mediawiki/php-master/includes/auth/AuthManager.php(532): MediaWiki\Auth\TemporaryPasswordPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
#18 /srv/mediawiki/php-master/includes/auth/AuthManager.php(462): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#19 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(374): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#20 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(506): AuthManagerSpecialPage->performAuthenticationStep(string, array)
#21 [internal function]: AuthManagerSpecialPage->handleFormSubmit(array, CodexHTMLForm)
#22 /srv/mediawiki/php-master/includes/htmlform/HTMLForm.php(748): call_user_func(array, array, CodexHTMLForm)
#23 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(437): HTMLForm->trySubmit()
#24 /srv/mediawiki/php-master/includes/specialpage/LoginSignupSpecialPage.php(317): AuthManagerSpecialPage->trySubmit()
#25 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(701): LoginSignupSpecialPage->execute(NULL)
#26 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(1564): SpecialPage->run(NULL)
#27 /srv/mediawiki/php-master/includes/MediaWiki.php(344): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#28 /srv/mediawiki/php-master/includes/MediaWiki.php(948): MediaWiki->performRequest()
#29 /srv/mediawiki/php-master/includes/MediaWiki.php(597): MediaWiki->main()
#30 /srv/mediawiki/php-master/index.php(50): MediaWiki->run()
#31 /srv/mediawiki/php-master/index.php(46): wfIndexMain()
#32 /srv/mediawiki/w/index.php(3): require(string)
#33 {main}

Details

Subject	Repo	Branch	Lines +/-
jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses	mediawiki/core	wmf/1.41.0-wmf.22	+46 -61
jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses	mediawiki/core	master	+46 -61
shell: Do not sandbox MediaWiki maintenance scripts	mediawiki/core	master	+3 -1
shell: Do not use a shell in firejail	mediawiki/core	master	+2 -0
shell: Always wrap maintenance scripts in mwscript	operations/mediawiki-config	master	+1 -1
shell: Add $wgShellMaintenanceRunner option	mediawiki/core	master	+66 -3

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Declined	Release	brennen	T340249 1.41.0-wmf.21 deployment blockers
Resolved	Release	dancy	T343724 1.41.0-wmf.22 deployment blockers
Resolved		tstarling	T343291 [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php

Event Timeline

Etonkovidova created this task.Aug 1 2023, 11:36 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 1 2023, 11:36 PM

Etonkovidova updated the task description. (Show Details)Aug 1 2023, 11:39 PM

hueitan subscribed.Aug 2 2023, 9:03 AM

TheresNoTime subscribed.Aug 2 2023, 9:19 AM

Little suspicious of rMW420f22226656: Make Shell::makeScriptCommand use run.php (935734)

Error

mwversion: 1.41.0-alpha
reqId: ZMob5kBUY-0MXePlnXmT9gAAABg
https://beta-logs.wmcloud.org/app/discover#/doc/5f0c9be0-0b6f-11ec-9cde-3f4490e09a26/logstash-deploy-1-7.0.0-1-2023.08.02?id=tB98tYkBhFLoKHIV_TP_

normalized_message

[{reqId}] {exception_url}   RuntimeException: Failed to run getConfiguration.php: No MWMultiVersion instance initialized! MWScript.php wrapper not used?

exception.trace

from /srv/mediawiki/php-master/includes/SiteConfiguration.php(608)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(115): SiteConfiguration->getConfig(string, array)
#1 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1725): MediaWiki\JobQueue\JobQueueGroupFactory::MediaWiki\JobQueue\{closure}(boolean, integer, array, NULL, array)
#2 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1555): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#3 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(117): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#4 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(150): MediaWiki\JobQueue\JobQueueGroupFactory->getCachedJobClasses(string)
#5 /srv/mediawiki/php-master/includes/user/UserGroupManagerFactory.php(122): MediaWiki\JobQueue\JobQueueGroupFactory->makeJobQueueGroup(string)
#6 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2747): MediaWiki\User\UserGroupManagerFactory->getUserGroupManager(string)
#7 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2575): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->localUserData(string)
#8 /srv/mediawiki/php-master/extensions/CentralAuth/includes/User/CentralAuthUser.php(2544): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->queryAttached()
#9 /srv/mediawiki/wmf-config/CommonSettings.php(2041): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->getLocalGroups()
#10 /srv/mediawiki/wmf-config/CommonSettings-labs.php(61): wmfGetPrivilegedGroups(User)
#11 /srv/mediawiki/php-master/includes/HookContainer/HookContainer.php(160): {closure}(User, array)
#12 /srv/mediawiki/php-master/includes/HookContainer/HookRunner.php(3065): MediaWiki\HookContainer\HookContainer->run(string, array)
#13 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(191): MediaWiki\HookContainer\HookRunner->onPasswordPoliciesForUser(User, array)
#14 /srv/mediawiki/php-master/includes/password/UserPasswordPolicy.php(81): UserPasswordPolicy->getPoliciesForUser(User)
#15 /srv/mediawiki/php-master/includes/user/User.php(1017): UserPasswordPolicy->checkUserPassword(User, string)
#16 /srv/mediawiki/php-master/includes/auth/AbstractPasswordPrimaryAuthenticationProvider.php(109): User->checkPasswordValidity(string)
#17 /srv/mediawiki/php-master/includes/auth/TemporaryPasswordPrimaryAuthenticationProvider.php(161): MediaWiki\Auth\AbstractPasswordPrimaryAuthenticationProvider->checkPasswordValidity(string, string)
#18 /srv/mediawiki/php-master/includes/auth/AuthManager.php(532): MediaWiki\Auth\TemporaryPasswordPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
#19 /srv/mediawiki/php-master/includes/auth/AuthManager.php(462): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#20 /srv/mediawiki/php-master/includes/api/ApiClientLogin.php(103): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#21 /srv/mediawiki/php-master/includes/api/ApiMain.php(1921): ApiClientLogin->execute()
#22 /srv/mediawiki/php-master/includes/api/ApiMain.php(898): ApiMain->executeAction()
#23 /srv/mediawiki/php-master/includes/api/ApiMain.php(869): ApiMain->executeActionWithErrorHandling()
#24 /srv/mediawiki/php-master/api.php(95): ApiMain->execute()
#25 /srv/mediawiki/php-master/api.php(48): wfApiMain()
#26 /srv/mediawiki/w/api.php(3): require(string)
#27 {main}

Impact

Notes

Ladsgroup subscribed.Aug 2 2023, 9:34 AM

Urbanecm_WMF subscribed.Aug 2 2023, 9:50 AM

Quiddity subscribed.Aug 2 2023, 4:09 PM

This also applies to temp users. When attempting to edit an article on https://de.wikipedia.beta.wmflabs.org/ as an anon user where it should create a temp user for IP masking, I get the same message:

[ZMqyjappClPgoVLMgOV8@gAAAA0] /w/index.php?action=submit&title=%F0%9F%87%AC%F0%9F%87%B1 RuntimeException: Failed to run getConfiguration.php: No MWMultiVersion instance initialized! MWScript.php wrapper not used?
Backtrace:

from /srv/mediawiki/php-master/includes/SiteConfiguration.php(608)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(115): SiteConfiguration->getConfig(string, array)
#1 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1725): MediaWiki\JobQueue\JobQueueGroupFactory::MediaWiki\JobQueue\{closure}(boolean, integer, array, NULL, array)
#2 /srv/mediawiki/php-master/includes/libs/objectcache/wancache/WANObjectCache.php(1555): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#3 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(117): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#4 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroupFactory.php(150): MediaWiki\JobQueue\JobQueueGroupFactory->getCachedJobClasses(string)
#5 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthUtilityService.php(169): MediaWiki\JobQueue\JobQueueGroupFactory->makeJobQueueGroup(string)
#6 /srv/mediawiki/php-master/extensions/CentralAuth/includes/Hooks/Handlers/UserCreationHookHandler.php(75): MediaWiki\Extension\CentralAuth\CentralAuthUtilityService->scheduleCreationJobs(MediaWiki\Extension\CentralAuth\User\CentralAuthUser)
#7 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(2183): MediaWiki\Extension\CentralAuth\Hooks\Handlers\UserCreationHookHandler->MediaWiki\Extension\CentralAuth\Hooks\Handlers\{closure}(integer, Wikimedia\Rdbms\DatabaseMysqli)
#8 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1574): Wikimedia\Rdbms\Database->runOnTransactionIdleCallbacks(integer, array)
#9 /srv/mediawiki/php-master/includes/libs/rdbms/lbfactory/LBFactory.php(382): Wikimedia\Rdbms\LoadBalancer->runPrimaryTransactionIdleCallbacks(string)
#10 /srv/mediawiki/php-master/includes/libs/rdbms/lbfactory/LBFactory.php(339): Wikimedia\Rdbms\LBFactory->executePostTransactionCallbacks()
#11 /srv/mediawiki/php-master/includes/MediaWiki.php(708): Wikimedia\Rdbms\LBFactory->commitPrimaryChanges(string, integer)
#12 /srv/mediawiki/php-master/includes/MediaWiki.php(678): MediaWiki::preOutputCommit(RequestContext)
#13 /srv/mediawiki/php-master/includes/MediaWiki.php(961): MediaWiki->doPreOutputCommit()
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(597): MediaWiki->main()
#15 /srv/mediawiki/php-master/index.php(50): MediaWiki->run()
#16 /srv/mediawiki/php-master/index.php(46): wfIndexMain()
#17 /srv/mediawiki/w/index.php(3): require(string)
#18 {main}

normalized_message

Executing: /bin/bash '/srv/mediawiki/php-master/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/bin/firejail'\'' '\''--quiet'\'' '\''--profile=/srv/mediawiki/php-master/includes/shell/firejail.profile'\'' '\''--noroot'\'' '\''--seccomp'\'' '\''--private-dev'\'' -- '\''/usr/bin/php'\'' '\''/srv/mediawiki/php-master/maintenance/run.php'\'' '\''getConfiguration'\'' '\''--wiki'\'' '\''arwiki'\'' '\''--settings'\'' '\''wgJobClasses'\'' '\''--format'\'' '\''PHP'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=50; SB_CGROUP='\''/sys/fs/cgroup/memory/mediawiki/job'\''; SB_MEM_LIMIT=0; SB_FILE_SIZE_LIMIT=0; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes'

Running

samtar@deployment-mwmaint02:~$ php /srv/mediawiki/php-master/maintenance/run.php getConfiguration --wiki arwiki

gives the reported error of No MWMultiVersion instance initialized! MWScript.php wrapper not used?

In comparison, running

samtar@deployment-mwmaint02:~$ mwscript getConfiguration --wiki arwiki --format json

returns as expected ({"wg3dProcessEnviron":{ ...)

rMWab802a66e685: jobqueue: Use a proper config system for dealing with settings (p.1) introduced getCachedJobClasses (with a note that "SiteConfiguration will have to be refactored to use our config system")

Revert of rMWab802a66e685: jobqueue: Use a proper config system for dealing with settings (p.1) submitted at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/944855

Temporarily doing

diff --git a/includes/jobqueue/JobQueueGroupFactory.php b/includes/jobqueue/JobQueueGroupFactory.php
index 680c1638782..cb8df364503 100644
--- a/includes/jobqueue/JobQueueGroupFactory.php
+++ b/includes/jobqueue/JobQueueGroupFactory.php
@@ -147,7 +147,7 @@ class JobQueueGroupFactory {
                        $this->instances[$domain] = new JobQueueGroup(
                                $domain,
                                $this->readOnlyMode,
-                               $this->getCachedJobClasses( $domain ),
+                               $this->options->get( MainConfigNames::JobClasses ),
                                $this->options->get( MainConfigNames::JobTypeConf ),
                                $this->options->get( MainConfigNames::JobTypesExcludedFromDefaultQueue ),
                                $this->statsdDataFactory,

resolved the issue, so its getCachedJobClasses() (or more accurately, $wgConf->getConfig() not handling multi-wiki stuff well)

I could be wrong, but as it stands without that revert, this'll break things when we roll to .21?

phuedx mentioned this in T343372: Selenium daily build tests failing with Node.js error.Aug 3 2023, 4:59 AM

Michael subscribed.Aug 3 2023, 8:50 AM

Michael mentioned this in T340832: [STORY] MUL - While reading, make Wikibase label fallbacks or the `mul` language code default clearly visible.Aug 3 2023, 8:53 AM

Jdlrobson subscribed.Aug 3 2023, 2:45 PM

I feel if $wgConf->getConfig() would be broken in production, we'd have noticed already.

Actually JobQueueGroupFactory::getCachedConfigVar() seems to be the only thing in existence calling it. Huh.

Jrbranaa subscribed.Aug 3 2023, 8:54 PM

• Samwalton9-WMF subscribed.Aug 4 2023, 12:49 PM

Novem_Linguae merged a task: T343532: (Beta Cluster) RuntimeException: Failed to run getConfiguration.php: No MWMultiVersion instance initialized.Aug 4 2023, 12:56 PM

Novem_Linguae subscribed.

Change 945878 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] shell: Add $wgShellMaintenanceRunner option

https://gerrit.wikimedia.org/r/945878

gerritbot added a project: Patch-For-Review.Aug 4 2023, 11:22 PM

Change 945878 abandoned by Gergő Tisza:

[mediawiki/core@master] shell: Add $wgShellMaintenanceRunner option

Reason:

https://gerrit.wikimedia.org/r/945878

Maintenance_bot removed a project: Patch-For-Review.Aug 4 2023, 11:30 PM

Change 945883 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] shell: Always wrap maintenance scripts in mwscript

https://gerrit.wikimedia.org/r/945883

gerritbot added a project: Patch-For-Review.Aug 4 2023, 11:43 PM

XenonX3 subscribed.Aug 5 2023, 9:12 PM

In T343291#9064640, @TheresNoTime wrote:

Temporarily doing

diff --git a/includes/jobqueue/JobQueueGroupFactory.php b/includes/jobqueue/JobQueueGroupFactory.php
index 680c1638782..cb8df364503 100644
--- a/includes/jobqueue/JobQueueGroupFactory.php
+++ b/includes/jobqueue/JobQueueGroupFactory.php
@@ -147,7 +147,7 @@ class JobQueueGroupFactory {
                        $this->instances[$domain] = new JobQueueGroup(
                                $domain,
                                $this->readOnlyMode,
-                               $this->getCachedJobClasses( $domain ),
+                               $this->options->get( MainConfigNames::JobClasses ),
                                $this->options->get( MainConfigNames::JobTypeConf ),
                                $this->options->get( MainConfigNames::JobTypesExcludedFromDefaultQueue ),
                                $this->statsdDataFactory,

resolved the issue, so its getCachedJobClasses() (or more accurately, $wgConf->getConfig() not handling multi-wiki stuff well)

You might know this already, but given that the proposed revert at change 944855 is still open, I'll write down a potential reason that this revert won't fix the issue, and that the above diff could cause a bug in production.

Both before and after change 937045, $wgConf->getConfig() is used to retreive $wgJobClasses for foreign wikis.

Before this change, as the revert restores, the value for the "current" wiki is injected from JobQueueGroupFactory to the JobQueueGroup parameter, which then subsequently ignores it for most wikis, and calls $wgConfig->getConfig() instead. After this change, JobQueueGroupFactory calls $wgConfig->getConfig() as-needed, and injects the correct value for JobQueueGroup to then safely use.

The above diff forces the local wiki value of wgJobClasses unto other wiki contexts, thus avoiding the wgConf call, but the full revert would not have avoided that call, I think.

Change 945883 merged by jenkins-bot:

[operations/mediawiki-config@master] shell: Always wrap maintenance scripts in mwscript

https://gerrit.wikimedia.org/r/945883

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2023, 8:10 AM

In T343291#9072561, @Krinkle wrote:
In T343291#9064640, @TheresNoTime wrote:
Temporarily doing
diff --git a/includes/jobqueue/JobQueueGroupFactory.php b/includes/jobqueue/JobQueueGroupFactory.php
index 680c1638782..cb8df364503 100644
--- a/includes/jobqueue/JobQueueGroupFactory.php
+++ b/includes/jobqueue/JobQueueGroupFactory.php
@@ -147,7 +147,7 @@ class JobQueueGroupFactory {
                        $this->instances[$domain] = new JobQueueGroup(
                                $domain,
                                $this->readOnlyMode,
-                               $this->getCachedJobClasses( $domain ),
+                               $this->options->get( MainConfigNames::JobClasses ),
                                $this->options->get( MainConfigNames::JobTypeConf ),
                                $this->options->get( MainConfigNames::JobTypesExcludedFromDefaultQueue ),
                                $this->statsdDataFactory,
resolved the issue, so its getCachedJobClasses() (or more accurately, $wgConf->getConfig() not handling multi-wiki stuff well)
You might know this already, but given that the proposed revert at change 944855 is still open, I'll write down a potential reason that this revert won't fix the issue, and that the above diff could cause a bug in production.

Both before and after change 937045, $wgConf->getConfig() is used to retreive $wgJobClasses for foreign wikis.

Before this change, as the revert restores, the value for the "current" wiki is injected from JobQueueGroupFactory to the JobQueueGroup parameter, which then subsequently ignores it for most wikis, and calls $wgConfig->getConfig() instead. After this change, JobQueueGroupFactory calls $wgConfig->getConfig() as-needed, and injects the correct value for JobQueueGroup to then safely use.

The above diff forces the local wiki value of wgJobClasses unto other wiki contexts, thus avoiding the wgConf call, but the full revert would not have avoided that call, I think.

I'll defer to your knowledge on this @Krinkle — either way, it'd be nice if we fixed this.. r/945883 above didn't resolve the issue

zeljkofilipin added a project: User-zeljkofilipin.Aug 7 2023, 12:14 PM

zeljkofilipin moved this task from Backlog 🪒 to Watching 📺 on the User-zeljkofilipin board.

zeljkofilipin subscribed.

It seems to have fixed the specific issue and triggered a new one:

tgr@deployment-mwmaint02:~$ mwscript shell.php aawiki

> $command = \MediaWiki\Shell\Shell::makeScriptCommand( 'getConfiguration', [ '--wiki', 'enwiki', '--settings', 'wgJobClasses', '--format', 'PHP' ] )

> (string)$command
= "#Command: '/usr/bin/php' '/srv/mediawiki/multiversion/MWScript.php' 'getConfiguration' '--wiki' 'enwiki' '--settings' 'wgJobClasses' '--format' 'PHP'"

> $result = $command->execute()
> $result->getStderr()
= "execvp: Permission denied\n"

This works just fine:

tgr@deployment-mwmaint02:~$ sudo -u www-data '/usr/bin/php' '/srv/mediawiki/multiversion/MWScript.php' 'getConfiguration' '--wiki' 'enwiki' '--settings' 'wgJobClasses' '--format' 'PHP'

Wrong user, or wrong firejail configuration, or something like that? Might or might not be an issue in production.

This works:

> $command = \MediaWiki\Shell\Shell::makeScriptCommand( 'getConfiguration', [ '--wiki', 'enwiki', '--settings', 'wgJobClasses', '--format', 'PHP' ] )
> $command->disableSandbox()->execute()->getStdOut()

This doesn't work:

> $command = \MediaWiki\Shell\Shell::makeScriptCommand( 'getConfiguration', [ '--wiki', 'enwiki', '--settings', 'wgJobClasses', '--format', 'PHP' ] )
> $command->restrict( 0 )->disableSandbox( false )->execute()->getStdOut()

so definitely Firejail-related.

The issue does affect production (I tested on mwmaint, not an appserver, but I doubt that would make a difference).

Unlike the previous error, this doesn't seem to be related to rMW420f22226656: Make Shell::makeScriptCommand use run.php btw:

> $command = \MediaWiki\Shell\Shell::makeScriptCommand( MW_INSTALL_PATH . '/maintenance/getConfiguration.php', [ '--wiki', 'enwiki', '--settings', 'wgJobClasses', '--format', 'PHP' ] )
> $command->execute()->getStderr()
= "execvp: Permission denied\n"

In T343291#9072561, @Krinkle wrote:

Both before and after change 937045, $wgConf->getConfig() is used to retreive $wgJobClasses for foreign wikis.

I guess the upshot is that this never actually worked and we just didn't notice? Unless something recently changed with Firejail or OS permissions configuration.

Tgr added a parent task: T343724: 1.41.0-wmf.22 deployment blockers.Aug 7 2023, 5:30 PM

The actual command that gets run is

/bin/bash '/srv/mediawiki/php-master/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/bin/firejail'\'' '\''--quiet'\'' '\''--profile=/srv/mediawiki/php-master/includes/shell/firejail.profile'\'' '\''--noroot'\'' '\''--seccomp'\'' '\''--private-dev'\'' -- '\''/usr/bin/php'\'' '\''/srv/mediawiki/multiversion/MWScript.php'\'' '\''getConfiguration'\'' '\''--wiki'\'' '\''enwiki'\'' '\''--settings'\'' '\''wgJobClasses'\'' '\''--format'\'' '\''PHP'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=50; SB_CGROUP='\''/sys/fs/cgroup/memory/mediawiki/job'\''; SB_MEM_LIMIT=1073741824; SB_FILE_SIZE_LIMIT=536870912; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes'⏎

A more minimal reproduction step is

sudo -u www-data /usr/bin/firejail --quiet --profile=/srv/mediawiki/php-master/includes/shell/firejail.profile -- /usr/bin/php /srv/mediawiki/w/health-check.php

In debug mode it says

Running '/usr/bin/php' '/srv/mediawiki/w/health-check.php'  command through /usr/sbin/nologin
execvp argument 0: /usr/sbin/nologin
execvp argument 1: -c
execvp argument 2: --
execvp argument 3: '/usr/bin/php' '/srv/mediawiki/w/health-check.php' 
Child process initialized in 71.05 ms
execvp: Permission denied

And indeed adding --shell=/bin/sh or --shell=none to the firejail command makes the script work.

Change 946607 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] shell: Do not use a shell in firejail

https://gerrit.wikimedia.org/r/946607

gerritbot added a project: Patch-For-Review.Aug 7 2023, 6:57 PM

Testing with the patch manually applied, I get

Warning: require(/srv/mediawiki/wmf-config/../private/PrivateSettings.php): failed to open stream: Permission denied in /srv/mediawiki/wmf-config/CommonSettings.php on line 319
Fatal error: require(): Failed opening required '/srv/mediawiki/wmf-config/../private/PrivateSettings.php' (include_path='/srv/mediawiki/php-master/vendor/pear/console_getopt:/srv/mediawiki/php-master/vendor/pear/mail:/srv/mediawiki/php-master/vendor/pear/mail_mime:/srv/mediawiki/php-master/vendor/pear/net_smtp:/srv/mediawiki/php-master/vendor/pear/net_socket:/srv/mediawiki/php-master/vendor/pear/net_url2:/srv/mediawiki/php-master/vendor/pear/pear-core-minimal/src:/srv/mediawiki/php-master/vendor/pear/pear_exception:/srv/mediawiki/php-master:/usr/local/lib/php:/usr/share/php') in /srv/mediawiki/wmf-config/CommonSettings.php on line 319

...which I think makes that a dead-end. We explicitly ban /srv/mediawiki/private access in /etc/firejail/mediawiki.local and it doesn't seem like a good idea to change that.

Arguably there isn't really any point to use firejail for running a MediaWiki maintenance script - if MediaWiki is compromised it can just circumvent firejail in the first place. So probably we should just not use it in this case.

Change 946607 abandoned by Gergő Tisza:

[mediawiki/core@master] shell: Do not use a shell in firejail

Reason:

on second thought probably not the right way to fix T343291

https://gerrit.wikimedia.org/r/946607

Change 946610 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/core@master] shell: Do not sandbox MediaWiki maintenance scripts

https://gerrit.wikimedia.org/r/946610

RhinosF1 merged a task: T343482: can not create account at wikifunctions.beta.wmflabs.org.Aug 8 2023, 6:41 AM

RhinosF1 added subscribers: Bugreporter, RhinosF1, 0xDeadbeef.

Change 946610 merged by jenkins-bot:

[mediawiki/core@master] shell: Do not sandbox MediaWiki maintenance scripts

https://gerrit.wikimedia.org/r/946610

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.22; 2023-08-15).Aug 8 2023, 11:00 AM

tstarling closed this task as Resolved.Aug 8 2023, 11:02 AM

tstarling claimed this task.

Maintenance_bot removed a project: Patch-For-Review.Aug 8 2023, 11:10 AM

Krinkle mentioned this in T344223: User logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found".Aug 15 2023, 11:47 AM

Change 949103 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949103

gerritbot added a project: Patch-For-Review.Aug 15 2023, 8:01 PM

Change 949178 had a related patch set uploaded (by D3r1ck01; author: Krinkle):

[mediawiki/core@wmf/1.41.0-wmf.22] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949178

Change 949103 merged by jenkins-bot:

[mediawiki/core@master] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949103

Change 949178 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.22] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949178

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:52:19Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]]

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:53:55Z] <urbanecm@deploy1002> urbanecm and d3r1ck01: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessibl

Mentioned in SAL (#wikimedia-operations) [2023-08-16T14:06:32Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] (duration: 14m 13s)