Page MenuHomePhabricator

User logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found"
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   RuntimeException: firejail is enabled, but cannot be found
exception.trace
from /srv/mediawiki/php-1.41.0-wmf.22/includes/shell/CommandFactory.php(125)
#0 /srv/mediawiki/php-1.41.0-wmf.22/includes/shell/CommandFactory.php(160): MediaWiki\Shell\CommandFactory->getLocalShellboxOptions()
#1 /srv/mediawiki/php-1.41.0-wmf.22/includes/shell/Shell.php(129): MediaWiki\Shell\CommandFactory->create()
#2 /srv/mediawiki/php-1.41.0-wmf.22/includes/shell/Shell.php(199): MediaWiki\Shell\Shell::command(array)
#3 /srv/mediawiki/php-1.41.0-wmf.22/includes/SiteConfiguration.php(599): MediaWiki\Shell\Shell::makeScriptCommand(string, array)
#4 /srv/mediawiki/php-1.41.0-wmf.22/includes/jobqueue/JobQueueGroupFactory.php(115): SiteConfiguration->getConfig(string, array)
#5 /srv/mediawiki/php-1.41.0-wmf.22/includes/libs/objectcache/wancache/WANObjectCache.php(1725): MediaWiki\JobQueue\JobQueueGroupFactory::MediaWiki\JobQueue\{closure}(boolean, integer, array, NULL, array)
#6 /srv/mediawiki/php-1.41.0-wmf.22/includes/libs/objectcache/wancache/WANObjectCache.php(1555): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#7 /srv/mediawiki/php-1.41.0-wmf.22/includes/jobqueue/JobQueueGroupFactory.php(117): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#8 /srv/mediawiki/php-1.41.0-wmf.22/includes/jobqueue/JobQueueGroupFactory.php(150): MediaWiki\JobQueue\JobQueueGroupFactory->getCachedJobClasses(string)
#9 /srv/mediawiki/php-1.41.0-wmf.22/includes/user/UserGroupManagerFactory.php(122): MediaWiki\JobQueue\JobQueueGroupFactory->makeJobQueueGroup(string)
#10 /srv/mediawiki/php-1.41.0-wmf.22/extensions/CentralAuth/includes/User/CentralAuthUser.php(2747): MediaWiki\User\UserGroupManagerFactory->getUserGroupManager(string)
#11 /srv/mediawiki/php-1.41.0-wmf.22/extensions/CentralAuth/includes/User/CentralAuthUser.php(2575): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->localUserData(string)
#12 /srv/mediawiki/php-1.41.0-wmf.22/extensions/CentralAuth/includes/User/CentralAuthUser.php(2544): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->queryAttached()
#13 /srv/mediawiki/wmf-config/CommonSettings.php(2041): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->getLocalGroups()
#14 /srv/mediawiki/wmf-config/CommonSettings.php(692): wmfGetPrivilegedGroups(User)
#15 /srv/mediawiki/php-1.41.0-wmf.22/includes/HookContainer/HookContainer.php(160): {closure}(User, array)
#16 /srv/mediawiki/php-1.41.0-wmf.22/includes/HookContainer/HookRunner.php(3065): MediaWiki\HookContainer\HookContainer->run(string, array)
#17 /srv/mediawiki/php-1.41.0-wmf.22/includes/password/UserPasswordPolicy.php(191): MediaWiki\HookContainer\HookRunner->onPasswordPoliciesForUser(User, array)
#18 /srv/mediawiki/php-1.41.0-wmf.22/includes/password/UserPasswordPolicy.php(81): UserPasswordPolicy->getPoliciesForUser(User)
#19 /srv/mediawiki/php-1.41.0-wmf.22/includes/user/User.php(999): UserPasswordPolicy->checkUserPassword(User, string)
#20 /srv/mediawiki/php-1.41.0-wmf.22/includes/auth/AbstractPasswordPrimaryAuthenticationProvider.php(109): User->checkPasswordValidity(string)
#21 /srv/mediawiki/php-1.41.0-wmf.22/includes/auth/TemporaryPasswordPrimaryAuthenticationProvider.php(161): MediaWiki\Auth\AbstractPasswordPrimaryAuthenticationProvider->checkPasswordValidity(string, string)
#22 /srv/mediawiki/php-1.41.0-wmf.22/includes/auth/AuthManager.php(557): MediaWiki\Auth\TemporaryPasswordPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
#23 /srv/mediawiki/php-1.41.0-wmf.22/includes/auth/AuthManager.php(487): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#24 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/AuthManagerSpecialPage.php(374): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#25 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/AuthManagerSpecialPage.php(506): AuthManagerSpecialPage->performAuthenticationStep(string, array)
#26 /srv/mediawiki/php-1.41.0-wmf.22/includes/htmlform/HTMLForm.php(748): AuthManagerSpecialPage->handleFormSubmit(array, CodexHTMLForm)
#27 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/AuthManagerSpecialPage.php(437): HTMLForm->trySubmit()
#28 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/LoginSignupSpecialPage.php(317): AuthManagerSpecialPage->trySubmit()
#29 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/SpecialPage.php(701): LoginSignupSpecialPage->execute(NULL)
#30 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/SpecialPageFactory.php(1565): SpecialPage->run(NULL)
#31 /srv/mediawiki/php-1.41.0-wmf.22/includes/MediaWiki.php(344): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#32 /srv/mediawiki/php-1.41.0-wmf.22/includes/MediaWiki.php(942): MediaWiki->performRequest()
#33 /srv/mediawiki/php-1.41.0-wmf.22/includes/MediaWiki.php(591): MediaWiki->main()
#34 /srv/mediawiki/php-1.41.0-wmf.22/index.php(50): MediaWiki->run()
#35 /srv/mediawiki/php-1.41.0-wmf.22/index.php(46): wfIndexMain()
#36 /srv/mediawiki/w/index.php(3): require(string)
#37 {main}
Impact
Notes

Triggered in front of me at Wikimania. Joy.

Event Timeline

Jdforrester-WMF created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

So the core feature of "load configuration of an another wiki" is broken here. I think that's because SiteConfiguration::getConfig tries to run a maintenance script to get the config for another wiki and that's broken in MW-in-K8s.

Possibly related: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/946610, which is in this week's train. Marking as a train blocker also.

taavi renamed this task from 2FA-enabled user logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found" to User logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found".Aug 15 2023, 10:41 AM
taavi raised the priority of this task from Medium to Unbreak Now!.

The same way of cross-wiki job queueing during login broke Beta two weeks ago, which led to the above fix: T343291: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php.

The main change was https://gerrit.wikimedia.org/r/c/mediawiki/core/+/937045, before that the SiteConfiguration was not called on JobQueueGroup creation.
It is now called in more situation (when job queue is involved, not only when the job queue is used. I am not sure, but in the old code it was used on job pop or other actions to read from the queue, not many code paths used the remote config, no one on writting to the queue)

Acknowledging that the train is blocked on this ticket.

Change 949103 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949103

Change 949178 had a related patch set uploaded (by D3r1ck01; author: Krinkle):

[mediawiki/core@wmf/1.41.0-wmf.22] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949178

Change 949103 merged by jenkins-bot:

[mediawiki/core@master] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949103

Change 949178 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.22] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949178

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:52:19Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]]

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:53:55Z] <urbanecm@deploy1002> urbanecm and d3r1ck01: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessibl

Mentioned in SAL (#wikimedia-operations) [2023-08-16T14:06:32Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] (duration: 14m 13s)

Hi @Krinkle. Can you give me a status report on this ticket?

Kappakayala subscribed.

Removing 'serviceops' tag as there is no work needed from ServiceOps team at this time. Please add the tag if there needs any help from SRE ServiceOps team.