Page MenuHomePhabricator
Create Task
Maniphest T367334

EntitySchema conditional namespace registration errors (NS_ENTITYSCHEMA_JSON, NamespaceRegistrationHandler::registerNamespace() TypeError)
Closed, ResolvedPublic5 Estimated Story PointsPRODUCTION ERROR
Actions

Description

This error was first observed when I tried to backport two changes to wmf.8 (quickly reverted); however, the error is still reproducible on Test Wikidata, so it’s not specific to the backports and blocks the train.

Steps to reproduce: take a tool that authenticates via OAuth; patch it to contact test.wikidata.org; try to log in. (I don’t know a tool that lets you select the “OAuth wiki” off the top of my head; I used my development version of Wikidata Lexeme Forms. If the broken code reaches Wikidata, then plenty of tools that target Wikidata start to break.)

Error
message
PHP Warning: Use of undefined constant NS_ENTITYSCHEMA_JSON - assumed 'NS_ENTITYSCHEMA_JSON' (this will throw an Error in a future version of PHP)
TypeError: Argument 2 passed to EntitySchema\MediaWiki\Hooks\NamespaceRegistrationHandler::registerNamespace() must be of the type int, string given, called in /srv/mediawiki/php-1.43.0-wmf.9/extensions/EntitySchema/src/MediaWiki/Hooks/NamespaceRegistrationHandler.php on line 52
trace
from /srv/mediawiki/php-1.43.0-wmf.9/extensions/EntitySchema/src/MediaWiki/Hooks/NamespaceRegistrationHandler.php(69)
#0 /srv/mediawiki/php-1.43.0-wmf.9/extensions/EntitySchema/src/MediaWiki/Hooks/NamespaceRegistrationHandler.php(52): EntitySchema\MediaWiki\Hooks\NamespaceRegistrationHandler->registerNamespace(array, string, string, boolean, boolean, string, boolean)
#1 /srv/mediawiki/php-1.43.0-wmf.9/includes/HookContainer/HookContainer.php(159): EntitySchema\MediaWiki\Hooks\NamespaceRegistrationHandler->onCanonicalNamespaces(array)
#2 /srv/mediawiki/php-1.43.0-wmf.9/includes/HookContainer/HookRunner.php(1037): MediaWiki\HookContainer\HookContainer->run(string, array)
#3 /srv/mediawiki/php-1.43.0-wmf.9/includes/title/NamespaceInfo.php(396): MediaWiki\HookContainer\HookRunner->onCanonicalNamespaces(array)
#4 /srv/mediawiki/php-1.43.0-wmf.9/includes/title/NamespaceInfo.php(422): MediaWiki\Title\NamespaceInfo->getCanonicalNamespaces()
#5 /srv/mediawiki/php-1.43.0-wmf.9/includes/language/Language.php(647): MediaWiki\Title\NamespaceInfo->getCanonicalIndex(string)
#6 /srv/mediawiki/php-1.43.0-wmf.9/includes/title/MediaWikiTitleCodec.php(421): Language->getNsIndex(string)
#7 /srv/mediawiki/php-1.43.0-wmf.9/includes/title/Title.php(2751): MediaWiki\Title\MediaWikiTitleCodec->splitTitleString(string, integer)
#8 /srv/mediawiki/php-1.43.0-wmf.9/includes/title/Title.php(456): MediaWiki\Title\Title->secureAndSplit(string, integer)
#9 /srv/mediawiki/php-1.43.0-wmf.9/includes/title/Title.php(404): MediaWiki\Title\Title::newFromTextThrow(string, integer)
#10 /srv/mediawiki/php-1.43.0-wmf.9/extensions/OAuth/src/SessionProvider.php(85): MediaWiki\Title\Title::newFromText(string)
#11 /srv/mediawiki/php-1.43.0-wmf.9/includes/session/SessionManager.php(542): MediaWiki\Extension\OAuth\SessionProvider->provideSessionInfo(MediaWiki\Request\WebRequest)
#12 /srv/mediawiki/php-1.43.0-wmf.9/includes/session/SessionManager.php(248): MediaWiki\Session\SessionManager->getSessionInfoForRequest(MediaWiki\Request\WebRequest)
#13 /srv/mediawiki/php-1.43.0-wmf.9/includes/Request/WebRequest.php(870): MediaWiki\Session\SessionManager->getSessionForRequest(MediaWiki\Request\WebRequest)
#14 /srv/mediawiki/php-1.43.0-wmf.9/includes/session/SessionManager.php(167): MediaWiki\Request\WebRequest->getSession()
#15 /srv/mediawiki/php-1.43.0-wmf.9/includes/Setup.php(488): MediaWiki\Session\SessionManager::getGlobalSession()
#16 /srv/mediawiki/php-1.43.0-wmf.9/includes/WebStart.php(85): require_once(string)
#17 /srv/mediawiki/php-1.43.0-wmf.9/index.php(50): require(string)
#18 /srv/mediawiki/w/index.php(3): require(string)
#19 {main}
Impact

“Pretty bad.” (Exact impact not totally clear to me, but certainly bad enough to block the train, I’m afraid.)

Notes

The error seems to be limited to Special:OAuth/initiate and CentralAuth:

lucaswerkmeister-wmde@mwlog2002:/srv/mw-log$ tail -n 500k error.log | grep NS_ENTITYSCHEMA_JSON | awk '{print $10}' | sort | uniq -c | sort -rn | sed -E 's/[0-9a-f]{39}/redacted/'
   1063 NS_ENTITYSCHEMA_JSON
   1054 /w/index.php?title=Special%3AOAuth%2Finitiate
      1 /w/api.php?centralauthtoken=redacted&origin=https%3A%2F%2Fru.wikipedia.org
      1 /w/api.php?centralauthtoken=redacted&origin=https%3A%2F%2Fru.wikipedia.org
      1 /w/api.php?centralauthtoken=redacted&origin=https%3A%2F%2Fru.wikipedia.org
      1 /w/api.php?centralauthtoken=redacted&origin=https%3A%2F%2Fru.wikipedia.org
      1 /w/api.php?centralauthtoken=redacted&origin=https%3A%2F%2Fru.wikipedia.org
      1 /w/api.php?centralauthtoken=redacted&origin=https%3A%2F%2Fru.wikipedia.org
      1 /w/api.php?centralauthtoken=redacted&origin=https%3A%2F%2Fru.wikipedia.org
      1 /w/api.php?centralauthtoken=redacted&origin=https%3A%2F%2Fru.wikipedia.org
      1 /w/api.php?centralauthtoken=redacted&origin=https%3A%2F%2Fru.wikipedia.org

There was also a different error during the deployment of those backports: “No such service: EntitySchema.EntitySchemaIsRepo”. That error seemed to go away after a brief spike, and I don’t know how related it is, nor whether it would also affect the train.

Details

Request URL
https://test.wikidata.org/w/index.php?title=Special%3AOAuth%2Finitiate
SubjectRepoBranchLines +/-
Call NamespaceRegistrationHandler::setConstants() earliermediawiki/extensions/EntitySchemawmf/1.43.0-wmf.9+5 -3
Call NamespaceRegistrationHandler::setConstants() earliermediawiki/extensions/EntitySchemamaster+5 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Lucas_Werkmeister_WMDE triaged this task as Medium priority.Jun 12 2024, 4:13 PM
Lucas_Werkmeister_WMDE created this task.
Restricted Application added a project: Wikimedia-production-error. · View Herald TranscriptJun 12 2024, 4:13 PM
Lucas_Werkmeister_WMDE added a parent task: T363153: [ES-M2]: Load EntitySchema data type registration for WikibaseClient on client wikis.Jun 12 2024, 4:15 PM
Restricted Application added a project: wmde-wikidata-tech. · View Herald TranscriptJun 12 2024, 4:15 PM
Lucas_Werkmeister_WMDE mentioned this in T363153: [ES-M2]: Load EntitySchema data type registration for WikibaseClient on client wikis.Jun 12 2024, 4:16 PM
Maintenance_bot moved this task from Incoming to Ongoing on the wmde-wikidata-tech board.Jun 12 2024, 4:29 PM
gerritbot added a comment.Jun 12 2024, 4:31 PM

Change #1042308 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/EntitySchema@master] Call NamespaceRegistrationHandler::setConstants() earlier

https://gerrit.wikimedia.org/r/1042308

gerritbot added a project: Patch-For-Review.Jun 12 2024, 4:31 PM
Lucas_Werkmeister_WMDE moved this task from In Task Breakdown to Ready for Peer Review on the Wikidata Dev Team (Wikidata.org Slice) board.Jun 12 2024, 4:31 PM
brennen raised the priority of this task from Medium to Unbreak Now!.Jun 12 2024, 4:32 PM

Raising to UBN per train blocker policy.

Lucas_Werkmeister_WMDE added a comment.EditedJun 12 2024, 4:41 PM

I was able to reproduce the OAuth issue locally and the above Gerrit change seems to fix it; I’m just hoping that that’s enough to fix the CentralAuth error as well (I don’t think I can test that locally). It’ll need backporting to wmf.9, and, if we want to try the wmf.8 backports again, to wmf.8 too (but see below).

There was also a different error during the deployment of those backports: “No such service: EntitySchema.EntitySchemaIsRepo”. That error seemed to go away after a brief spike, and I don’t know how related it is, nor whether it would also affect the train.

This one is probably just a race condition – PHP processes with the old code started to see the new extension.json file (which injects the new EntitySchema.EntitySchemaIsRepo service into a bunch of hook handlers) before the php-fpm-restart let them see the new service definition in EntitySchema.ServiceWiring.php. (Edit: I can also confirm by checking the logs that this error seems to have been limited to bare-metal mw[0-9]{4} hosts; in mw-on-k8s, you wouldn’t expect this error to happen, as the new image gets deployed all at once: the new extension.json file is only seen by the new PHP code.) If we want to try backporting these changes to wmf.8 again, we’ll have to keep that in mind; but right now I think I’d rather not do that, and just wait until next week before doing T332157.

Stashbot mentioned this in T361403: 1.43.0-wmf.9 deployment blockers.Jun 12 2024, 6:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-12T18:01:47Z] <brennen> 1.43.0-wmf.9 train (T361403): currently blocked on T367334, holding at group0 until resolved.

brennen added a comment.Jun 12 2024, 6:06 PM

For clarity, is this breakage actually new in some way for wmf.9? I see quite a few of the errors in wmf.8. If nothing changes with the rollout, then this probably shouldn't block.

gerritbot added a comment.Jun 12 2024, 6:33 PM

Change #1042308 merged by jenkins-bot:

[mediawiki/extensions/EntitySchema@master] Call NamespaceRegistrationHandler::setConstants() earlier

https://gerrit.wikimedia.org/r/1042308

gerritbot added a comment.Jun 12 2024, 6:35 PM

Change #1042343 had a related patch set uploaded (by Brennen Bearnes; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/EntitySchema@wmf/1.43.0-wmf.9] Call NamespaceRegistrationHandler::setConstants() earlier

https://gerrit.wikimedia.org/r/1042343

gerritbot added a comment.Jun 12 2024, 7:32 PM

Change #1042343 merged by jenkins-bot:

[mediawiki/extensions/EntitySchema@wmf/1.43.0-wmf.9] Call NamespaceRegistrationHandler::setConstants() earlier

https://gerrit.wikimedia.org/r/1042343

Stashbot added a comment.Jun 12 2024, 7:32 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-12T19:32:31Z] <brennen@deploy1002> Started scap: Backport for [[gerrit:1042343|Call NamespaceRegistrationHandler::setConstants() earlier (T367334 T363153)]]

Stashbot added a comment.Jun 12 2024, 7:35 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-12T19:35:12Z] <brennen@deploy1002> brennen: Backport for [[gerrit:1042343|Call NamespaceRegistrationHandler::setConstants() earlier (T367334 T363153)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Jun 12 2024, 7:45 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-12T19:45:39Z] <brennen@deploy1002> Finished scap: Backport for [[gerrit:1042343|Call NamespaceRegistrationHandler::setConstants() earlier (T367334 T363153)]] (duration: 13m 06s)

brennen lowered the priority of this task from Unbreak Now! to Medium.Jun 12 2024, 7:46 PM
brennen removed a parent task: T361403: 1.43.0-wmf.9 deployment blockers.
brennen moved this task from Backlog to Logs/Train on the User-brennen board.Jun 12 2024, 8:16 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 12 2024, 8:31 PM
ArthurTaylor moved this task from Ready for Peer Review to Ready for Tech Verification on the Wikidata Dev Team (Wikidata.org Slice) board.Jun 13 2024, 8:19 AM
Lucas_Werkmeister_WMDE added a comment.EditedJun 13 2024, 9:13 AM
In T367334#9886017, @brennen wrote:

For clarity, is this breakage actually new in some way for wmf.9? I see quite a few of the errors in wmf.8. If nothing changes with the rollout, then this probably shouldn't block.

That would be because I tried to backport the broken patches to wmf.8, then reverted when the errors happened. Or did you see the wmf.8 errors outside of ca. 14:14–15:00 UTC yesterday?

Lucas_Werkmeister_WMDE closed this task as Resolved.Jun 13 2024, 11:09 AM
Lucas_Werkmeister_WMDE claimed this task.
Lucas_Werkmeister_WMDE moved this task from Ready for Tech Verification to Done on the Wikidata Dev Team (Wikidata.org Slice) board.

I think we can close this: the fix was merged and backported, the train rolled forward (group1 is on wmf.9 now), and the errors didn’t recur as far as I can tell.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits