Page MenuHomePhabricator
Create Task
Maniphest T344223

User logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found"
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   RuntimeException: firejail is enabled, but cannot be found
exception.trace
from /srv/mediawiki/php-1.41.0-wmf.22/includes/shell/CommandFactory.php(125)
#0 /srv/mediawiki/php-1.41.0-wmf.22/includes/shell/CommandFactory.php(160): MediaWiki\Shell\CommandFactory->getLocalShellboxOptions()
#1 /srv/mediawiki/php-1.41.0-wmf.22/includes/shell/Shell.php(129): MediaWiki\Shell\CommandFactory->create()
#2 /srv/mediawiki/php-1.41.0-wmf.22/includes/shell/Shell.php(199): MediaWiki\Shell\Shell::command(array)
#3 /srv/mediawiki/php-1.41.0-wmf.22/includes/SiteConfiguration.php(599): MediaWiki\Shell\Shell::makeScriptCommand(string, array)
#4 /srv/mediawiki/php-1.41.0-wmf.22/includes/jobqueue/JobQueueGroupFactory.php(115): SiteConfiguration->getConfig(string, array)
#5 /srv/mediawiki/php-1.41.0-wmf.22/includes/libs/objectcache/wancache/WANObjectCache.php(1725): MediaWiki\JobQueue\JobQueueGroupFactory::MediaWiki\JobQueue\{closure}(boolean, integer, array, NULL, array)
#6 /srv/mediawiki/php-1.41.0-wmf.22/includes/libs/objectcache/wancache/WANObjectCache.php(1555): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#7 /srv/mediawiki/php-1.41.0-wmf.22/includes/jobqueue/JobQueueGroupFactory.php(117): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#8 /srv/mediawiki/php-1.41.0-wmf.22/includes/jobqueue/JobQueueGroupFactory.php(150): MediaWiki\JobQueue\JobQueueGroupFactory->getCachedJobClasses(string)
#9 /srv/mediawiki/php-1.41.0-wmf.22/includes/user/UserGroupManagerFactory.php(122): MediaWiki\JobQueue\JobQueueGroupFactory->makeJobQueueGroup(string)
#10 /srv/mediawiki/php-1.41.0-wmf.22/extensions/CentralAuth/includes/User/CentralAuthUser.php(2747): MediaWiki\User\UserGroupManagerFactory->getUserGroupManager(string)
#11 /srv/mediawiki/php-1.41.0-wmf.22/extensions/CentralAuth/includes/User/CentralAuthUser.php(2575): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->localUserData(string)
#12 /srv/mediawiki/php-1.41.0-wmf.22/extensions/CentralAuth/includes/User/CentralAuthUser.php(2544): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->queryAttached()
#13 /srv/mediawiki/wmf-config/CommonSettings.php(2041): MediaWiki\Extension\CentralAuth\User\CentralAuthUser->getLocalGroups()
#14 /srv/mediawiki/wmf-config/CommonSettings.php(692): wmfGetPrivilegedGroups(User)
#15 /srv/mediawiki/php-1.41.0-wmf.22/includes/HookContainer/HookContainer.php(160): {closure}(User, array)
#16 /srv/mediawiki/php-1.41.0-wmf.22/includes/HookContainer/HookRunner.php(3065): MediaWiki\HookContainer\HookContainer->run(string, array)
#17 /srv/mediawiki/php-1.41.0-wmf.22/includes/password/UserPasswordPolicy.php(191): MediaWiki\HookContainer\HookRunner->onPasswordPoliciesForUser(User, array)
#18 /srv/mediawiki/php-1.41.0-wmf.22/includes/password/UserPasswordPolicy.php(81): UserPasswordPolicy->getPoliciesForUser(User)
#19 /srv/mediawiki/php-1.41.0-wmf.22/includes/user/User.php(999): UserPasswordPolicy->checkUserPassword(User, string)
#20 /srv/mediawiki/php-1.41.0-wmf.22/includes/auth/AbstractPasswordPrimaryAuthenticationProvider.php(109): User->checkPasswordValidity(string)
#21 /srv/mediawiki/php-1.41.0-wmf.22/includes/auth/TemporaryPasswordPrimaryAuthenticationProvider.php(161): MediaWiki\Auth\AbstractPasswordPrimaryAuthenticationProvider->checkPasswordValidity(string, string)
#22 /srv/mediawiki/php-1.41.0-wmf.22/includes/auth/AuthManager.php(557): MediaWiki\Auth\TemporaryPasswordPrimaryAuthenticationProvider->beginPrimaryAuthentication(array)
#23 /srv/mediawiki/php-1.41.0-wmf.22/includes/auth/AuthManager.php(487): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#24 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/AuthManagerSpecialPage.php(374): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#25 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/AuthManagerSpecialPage.php(506): AuthManagerSpecialPage->performAuthenticationStep(string, array)
#26 /srv/mediawiki/php-1.41.0-wmf.22/includes/htmlform/HTMLForm.php(748): AuthManagerSpecialPage->handleFormSubmit(array, CodexHTMLForm)
#27 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/AuthManagerSpecialPage.php(437): HTMLForm->trySubmit()
#28 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/LoginSignupSpecialPage.php(317): AuthManagerSpecialPage->trySubmit()
#29 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/SpecialPage.php(701): LoginSignupSpecialPage->execute(NULL)
#30 /srv/mediawiki/php-1.41.0-wmf.22/includes/specialpage/SpecialPageFactory.php(1565): SpecialPage->run(NULL)
#31 /srv/mediawiki/php-1.41.0-wmf.22/includes/MediaWiki.php(344): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#32 /srv/mediawiki/php-1.41.0-wmf.22/includes/MediaWiki.php(942): MediaWiki->performRequest()
#33 /srv/mediawiki/php-1.41.0-wmf.22/includes/MediaWiki.php(591): MediaWiki->main()
#34 /srv/mediawiki/php-1.41.0-wmf.22/index.php(50): MediaWiki->run()
#35 /srv/mediawiki/php-1.41.0-wmf.22/index.php(46): wfIndexMain()
#36 /srv/mediawiki/w/index.php(3): require(string)
#37 {main}
Impact
Notes

Triggered in front of me at Wikimania. Joy.

Details

Request URL
https://test.wikidata.org/w/index.php?returnto=*&title=*
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClassesmediawiki/corewmf/1.41.0-wmf.22+46 -61
jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClassesmediawiki/coremaster+46 -61
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jdforrester-WMF created this task.Aug 15 2023, 6:09 AM
Jdforrester-WMF triaged this task as Medium priority.
Restricted Application added a project: MediaWiki-Platform-Team. · View Herald TranscriptAug 15 2023, 6:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Novem_Linguae subscribed.Aug 15 2023, 6:10 AM
Chlod subscribed.Aug 15 2023, 6:12 AM
taavi subscribed.Aug 15 2023, 10:30 AM

So the core feature of "load configuration of an another wiki" is broken here. I think that's because SiteConfiguration::getConfig tries to run a maintenance script to get the config for another wiki and that's broken in MW-in-K8s.

taavi added a comment.Aug 15 2023, 10:41 AM

Possibly related: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/946610, which is in this week's train. Marking as a train blocker also.

taavi renamed this task from 2FA-enabled user logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found" to User logging in on mw-on-k8s triggers "RuntimeException: firejail is enabled, but cannot be found".Aug 15 2023, 10:41 AM
taavi raised the priority of this task from Medium to Unbreak Now!.
taavi added a parent task: T343724: 1.41.0-wmf.22 deployment blockers.
taavi edited projects, added MediaWiki-Shell, Shellbox; removed MediaWiki-extensions-OATHAuth, MediaWiki-extensions-CentralAuth, MediaWiki-User-login-and-signup.Aug 15 2023, 10:52 AM
Krinkle added a project: serviceops.Aug 15 2023, 11:39 AM
Krinkle subscribed.Aug 15 2023, 11:47 AM

The same way of cross-wiki job queueing during login broke Beta two weeks ago, which led to the above fix: T343291: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php.

Umherirrender subscribed.Aug 15 2023, 12:18 PM

The main change was https://gerrit.wikimedia.org/r/c/mediawiki/core/+/937045, before that the SiteConfiguration was not called on JobQueueGroup creation.
It is now called in more situation (when job queue is involved, not only when the job queue is used. I am not sure, but in the old code it was used on job pop or other actions to read from the queue, not many code paths used the remote config, no one on writting to the queue)

Novem_Linguae updated the task description. (Show Details)Aug 15 2023, 1:15 PM
ArielGlenn subscribed.Aug 15 2023, 1:24 PM
dancy subscribed.Aug 15 2023, 6:03 PM

Acknowledging that the train is blocked on this ticket.

Krinkle claimed this task.Aug 15 2023, 6:48 PM
Krinkle moved this task from Inbox, needs triage to In progress (DO NOT USE) on the MediaWiki-Platform-Team board.
gerritbot added a comment.Aug 15 2023, 7:50 PM

Change 949103 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949103

gerritbot added a project: Patch-For-Review.Aug 15 2023, 7:50 PM
gerritbot added a comment.Aug 16 2023, 8:42 AM

Change 949178 had a related patch set uploaded (by D3r1ck01; author: Krinkle):

[mediawiki/core@wmf/1.41.0-wmf.22] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949178

gerritbot added a comment.Aug 16 2023, 8:52 AM

Change 949103 merged by jenkins-bot:

[mediawiki/core@master] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949103

gerritbot added a comment.Aug 16 2023, 1:43 PM

Change 949178 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.22] jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses

https://gerrit.wikimedia.org/r/949178

Stashbot mentioned this in T344143: New Impact module: Run backend updating logic on all Wikipedias.Aug 16 2023, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:52:19Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]]

Stashbot mentioned this in T343291: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php.Aug 16 2023, 1:52 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-16T13:53:55Z] <urbanecm@deploy1002> urbanecm and d3r1ck01: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessibl

Stashbot added a comment.Aug 16 2023, 2:06 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-16T14:06:32Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949178|jobqueue: Disallow cross-wiki JobQueueGroup calls that require JobClasses (T344223 T343291)]], [[gerrit:949033|Growth: Enable new Impact backend on large Wikipedias (T344143)]] (duration: 14m 13s)

Maintenance_bot removed a project: Patch-For-Review.Aug 16 2023, 2:10 PM
Michael subscribed.Aug 16 2023, 2:15 PM
• Manuel mentioned this in T312097: [EPIC] MUL - Default values for labels and aliases .Aug 16 2023, 2:15 PM
dancy added a comment.Aug 16 2023, 2:39 PM

Hi @Krinkle. Can you give me a status report on this ticket?

Krinkle closed this task as Resolved.Aug 16 2023, 4:10 PM

Fixed.

Screenshot 2023-08-16 at 17.10.39.png (972×2 px, 208 KB)

dancy added a comment.Aug 16 2023, 4:16 PM

Thanks @Krinkle !

Kappakayala removed a project: serviceops.Aug 18 2023, 5:11 PM
Kappakayala subscribed.

Removing 'serviceops' tag as there is no work needed from ServiceOps team at this time. Please add the tag if there needs any help from SRE ServiceOps team.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits