Page MenuHomePhabricator

Wikibase CI broken with several errors
Closed, ResolvedPublic

Description

Wikibase CI is broken with various errors, e.g. empty change.

A WikibaseQualityConstraints PHPUnit test is failing with the following error message:

16:20:51 1) WikibaseQuality\ConstraintReport\Tests\Api\CheckConstraintsRdfTest::testShow
16:20:51 MediaWiki\Request\WebResponse::statusHeader(404) was not expected to be called.
16:20:51 
16:20:51 /workspace/src/extensions/WikibaseQualityConstraints/src/Api/CheckConstraintsRdf.php:132
16:20:51 /workspace/src/extensions/WikibaseQualityConstraints/tests/phpunit/Api/CheckConstraintsRdfTest.php:184
16:20:51 phpvfscomposer:///workspace/src/vendor/phpunit/phpunit/phpunit:106

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium/75108/console

The selenium job fails with several errors similar to the following one:

16:20:52 Error: internal_api_error_MWUnknownContentModelException: [7e0ed22118838965bda53589] Exception caught: The content model 'wikibase-item' is not registered on this wiki. See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium/72197/console

The apitests job fails with several errors similar to the following one:

16:19:17      AssertionError: User "Alice3zGO1CxLFm": Action "wbeditentity" returned error code "internal_api_error_MWUnknownContentModelException": [4c67cd37941eb99f1577c87e] Exception caught: The content model 'wikibase-item' is not registered on this wiki. See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.!

https://integration.wikimedia.org/ci/job/mediawiki-quibble-apitests-vendor-php74/36609/console

Event Timeline

Might be caused by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1115497, as the backport of that change started to show the same errors on mwdebug (logstash):

The content model 'wikibase-item' is not registered on this wiki.
from /srv/mediawiki/php-1.44.0-wmf.17/includes/content/ContentHandlerFactory.php(186)

Though it’s not yet clear how they could be related.

It looks like this is another occurrence of T288819: NamespaceInfo service missing namespaces if initialized too early. According to @Tgr (IRC logs), the CentralAuth patch adds two dependencies to its SetupAfterCache hook handler (Wikibase also uses this hook handler to register its content models, in RepoHooks::onSetupAfterCache()), PreferencesFactory and UserNameUtils; UserNameUtils ends up pulling in NamespaceInfo (UserNameUtilsContentLanguageLanguageFactoryNamespaceInfo), and so now when Wikibase’s hook handler runs, NamespaceInfo has already been created and will no longer see the changes that Wikibase makes to the globals.

RepoHooks::onSetupAfterCache()

Can it be done in an earlier hook? Generally, the only hook where changing globals is safe is MediaWikiServices (but you can't access services there).

FTR, the impact of this is basically complete site breakage. We’re fortunate that this was noticed on mwdebug. (The CentralAuth patch has been reverted on the wmf branches for now, though not yet on master, so Wikibase CI probably remains broken.)

image.png (271×1 px, 56 KB)

RepoHooks::onSetupAfterCache()

Can it be done in an earlier hook? Generally, the only hook where changing globals is safe is MediaWikiServices (but you can't access services there).

Probably – we should look into that.

Example of the error in production (on mwdebug):

The content model 'wikibase-item' is not registered on this wiki.
See https://www.mediawiki.org/wiki/Content_handlers to find out which extensions handle this content model.
from /srv/mediawiki/php-1.44.0-wmf.17/includes/content/ContentHandlerFactory.php(186)
#0 /srv/mediawiki/php-1.44.0-wmf.17/includes/content/ContentHandlerFactory.php(241): MediaWiki\Content\ContentHandlerFactory->validateContentHandler(string, null)
#1 /srv/mediawiki/php-1.44.0-wmf.17/includes/content/ContentHandlerFactory.php(175): MediaWiki\Content\ContentHandlerFactory->createContentHandlerFromHook(string)
#2 /srv/mediawiki/php-1.44.0-wmf.17/includes/content/ContentHandlerFactory.php(92): MediaWiki\Content\ContentHandlerFactory->createForModelID(string)
#3 /srv/mediawiki/php-1.44.0-wmf.17/includes/page/WikiPage.php(245): MediaWiki\Content\ContentHandlerFactory->getContentHandler(string)
#4 /srv/mediawiki/php-1.44.0-wmf.17/includes/actions/ActionEntryPoint.php(629): WikiPage->getContentHandler()
#5 /srv/mediawiki/php-1.44.0-wmf.17/includes/actions/ActionEntryPoint.php(507): MediaWiki\Actions\ActionEntryPoint->initializeArticle()
#6 /srv/mediawiki/php-1.44.0-wmf.17/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
#7 /srv/mediawiki/php-1.44.0-wmf.17/includes/MediaWikiEntryPoint.php(202): MediaWiki\Actions\ActionEntryPoint->execute()
#8 /srv/mediawiki/php-1.44.0-wmf.17/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#9 /srv/mediawiki/w/index.php(3): require(string)
#10 {main}

(logstash)

Generally, the only hook where changing globals is safe is MediaWikiServices (but you can't access services there).

Well, that’s going to be a problem, as the information about which namespaces we want to register currently lives completely inside the service container (WikibaseRepo.Settings, WikibaseRepo.LocalEntitySource, WikibaseRepo.EntityNamespaceLookup, WikibaseRepo.ContentModelMappings) :/

I feel like it might be better to introduce a way for Wikibase to tell the other services about the new namespaces, though that’s probably also not easy (for a start: which services?).

Change #1121043 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/CentralAuth@master] Revert "Add configuration options and global preference for the SUL3 rolllout"

https://gerrit.wikimedia.org/r/1121043

It looks like this is another occurrence of T288819: NamespaceInfo service missing namespaces if initialized too early. According to @Tgr (IRC logs), the CentralAuth patch adds two dependencies to its SetupAfterCache hook handler (Wikibase also uses this hook handler to register its content models, in RepoHooks::onSetupAfterCache()), PreferencesFactory and UserNameUtils; UserNameUtils ends up pulling in NamespaceInfo (UserNameUtilsContentLanguageLanguageFactoryNamespaceInfo), and so now when Wikibase’s hook handler runs, NamespaceInfo has already been created and will no longer see the changes that Wikibase makes to the globals.

The new code doesn't need most of UserNameUtils; it only calls isTemp() and isIP(), both of which can be implemented using narrower services (or static code). Would that be enough to resolve this, at least for now?

Change #1121044 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] WIP: Use MediaWikiServices hook

https://gerrit.wikimedia.org/r/1121044

The new code doesn't need most of UserNameUtils; it only calls isTemp() and isIP(), both of which can be implemented using narrower services (or static code). Would that be enough to resolve this, at least for now?

I think so, yeah – should be straightforward enough to test in CI using a Wikibase change that Depends-On the CentralAuth change.

Change #1121047 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/CentralAuth@master] [WIP] SharedDomainUtils: Avoid UserNameUtils service

https://gerrit.wikimedia.org/r/1121047

The other short-term fix we could do is split up SharedDomainUtils and SharedDomainHookHandler. The SetupAfterCache hook doesn't really have dependencies other than a bunch of config settings.

I think the proper solution would be to use a hook (or create one if needed) that's specifically for dynamically registering namespaces. The globals in question are $wgNamespaceContentModels and $wgContentHandlers (I think this breakage was actually caused by the latter). Per-namespace default content models can be dynamically registered via the ContentHandlerDefaultModelFor hook, and content handlers via ContentHandlerForModelID.

I think the proper solution would be to use a hook (or create one if needed) that's specifically for dynamically registering namespaces. The globals in question are $wgNamespaceContentModels and $wgContentHandlers (I think this breakage was actually caused by the latter). Per-namespace default content models can be dynamically registered via the ContentHandlerDefaultModelFor hook, and content handlers via ContentHandlerForModelID.

Yeah, that also sounds good to me. (Note that RepoHooks also updates $wgExtraNamespaces and $wgNamespacesToBeSearchedDefault.)

Change #1121044 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] WIP: Use MediaWikiServices hook

https://gerrit.wikimedia.org/r/1121044

This change seems to work in CI so far – I’m interested what other people think about it.

Change #1121047 abandoned by Bartosz Dziewoński:

[mediawiki/extensions/CentralAuth@master] [WIP] SharedDomainUtils: Avoid UserNameUtils service

Reason:

PreferencesFactory also depends on NamespaceInfo, so this would require more work. The Wikibase patch looks more promising.

https://gerrit.wikimedia.org/r/1121047

Change #1121043 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Revert "Add configuration options and global preference for the SUL3 rolllout"

https://gerrit.wikimedia.org/r/1121043

Change #1121072 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@master] Restore "Add configuration options and global preference for the SUL3 rolllout"

https://gerrit.wikimedia.org/r/1121072

Change #1121047 restored by Gergő Tisza:

[mediawiki/extensions/CentralAuth@master] [WIP] SharedDomainUtils: Avoid UserNameUtils service

https://gerrit.wikimedia.org/r/1121047

Change #1121072 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Restore "Add configuration options and global preference for the SUL3 rolllout"

https://gerrit.wikimedia.org/r/1121072

Change #1121047 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] SharedDomainUtils: Avoid early instantiation of NamespaceInfo

https://gerrit.wikimedia.org/r/1121047

Change #1121328 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.16] Restore "Add configuration options and global preference for the SUL3 rolllout"

https://gerrit.wikimedia.org/r/1121328

Change #1121329 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.17] Restore "Add configuration options and global preference for the SUL3 rolllout"

https://gerrit.wikimedia.org/r/1121329

Change #1121330 had a related patch set uploaded (by Gergő Tisza; author: Bartosz Dziewoński):

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.16] SharedDomainUtils: Avoid early instantiation of NamespaceInfo

https://gerrit.wikimedia.org/r/1121330

Change #1121332 had a related patch set uploaded (by Gergő Tisza; author: Bartosz Dziewoński):

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.17] SharedDomainUtils: Avoid early instantiation of NamespaceInfo

https://gerrit.wikimedia.org/r/1121332

I'll try to deploy the CentralAuth change again today in the 14UTC window. CI is passing now but let me know if I should test anything Wikidata-specific during the deploy.

Change #1121328 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.16] Restore "Add configuration options and global preference for the SUL3 rolllout"

https://gerrit.wikimedia.org/r/1121328

Change #1121330 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.16] SharedDomainUtils: Avoid early instantiation of NamespaceInfo

https://gerrit.wikimedia.org/r/1121330

Change #1121329 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.17] Restore "Add configuration options and global preference for the SUL3 rolllout"

https://gerrit.wikimedia.org/r/1121329

Change #1121332 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.44.0-wmf.17] SharedDomainUtils: Avoid early instantiation of NamespaceInfo

https://gerrit.wikimedia.org/r/1121332

Mentioned in SAL (#wikimedia-operations) [2025-02-20T14:52:08Z] <ihurbain@deploy2002> Started scap sync-world: Backport for [[gerrit:1121328|Restore "Add configuration options and global preference for the SUL3 rolllout" (T386836)]], [[gerrit:1121329|Restore "Add configuration options and global preference for the SUL3 rolllout" (T386836)]], [[gerrit:1121330|SharedDomainUtils: Avoid early instantiation of NamespaceInfo (T386836)]], [[gerrit:1121332|SharedDomainUtils: Avoid early in

Mentioned in SAL (#wikimedia-operations) [2025-02-20T14:55:07Z] <ihurbain@deploy2002> tgr, ihurbain: Backport for [[gerrit:1121328|Restore "Add configuration options and global preference for the SUL3 rolllout" (T386836)]], [[gerrit:1121329|Restore "Add configuration options and global preference for the SUL3 rolllout" (T386836)]], [[gerrit:1121330|SharedDomainUtils: Avoid early instantiation of NamespaceInfo (T386836)]], [[gerrit:1121332|SharedDomainUtils: Avoid early instantiatio

Mentioned in SAL (#wikimedia-operations) [2025-02-20T15:28:11Z] <ihurbain@deploy2002> Finished scap sync-world: Backport for [[gerrit:1121328|Restore "Add configuration options and global preference for the SUL3 rolllout" (T386836)]], [[gerrit:1121329|Restore "Add configuration options and global preference for the SUL3 rolllout" (T386836)]], [[gerrit:1121330|SharedDomainUtils: Avoid early instantiation of NamespaceInfo (T386836)]], [[gerrit:1121332|SharedDomainUtils: Avoid early i

Change #1121044 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Replace SetupAfterCache hook with MediaWikiServices

https://gerrit.wikimedia.org/r/1121044

I don’t think there’s much to verify here, but let’s leave it open until the end of next week in case it blows up anything in production. (I think if it does cause errors they’re likely to be quite loud and obvious, so if we didn’t notice anything then it’s probably fine.)

Re-adding this to MW Platform inbox as a reminder to clean up the CentralAuth workaround (in two weeks or so).

I think from the Wikidata side we can call this done (the train has rolled out and I didn’t hear of any related breakage) – leaving the task open for CentralAuth.

Change #1127919 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@master] Fix SharedDomainUtils constructor signature

https://gerrit.wikimedia.org/r/1127919

Change #1127919 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Fix SharedDomainUtils constructor signature

https://gerrit.wikimedia.org/r/1127919