Page MenuHomePhabricator

SimpleCacheWithBagOStuff: Cache key contains characters that are not allowed
Closed, DuplicatePublicPRODUCTION ERROR

Description

Error message
[XmlSFQpAADsAAAbbwwoAAADQ] /wiki/Category:Subcultures?uselang=%E2%A7%BCLang%E2%A7%BD   Wikibase\Lib\CacheInvalidArgumentException from line 264 of /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/SimpleCacheWithBagOStuff.php: Cache key contains characters that are not allowed: `Q264965_1128751259_⧼lang⧽_label`
Impact

A handful of these since 1.35.0-wmf.23 (T233871) rollout to group1.

Notes

See also, maybe: T244562

Details

Request ID
XmlSFQpAADsAAAbbwwoAAADQ
Request URL
https://commons.wikimedia.org/wiki/Category:Subcultures?uselang=%E2%A7%BCLang%E2%A7%BD
Stack Trace
#0 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/SimpleCacheWithBagOStuff.php(69): Wikibase\Lib\SimpleCacheWithBagOStuff->assertKeyIsValid(string)
#1 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/StatsdRecordingSimpleCache.php(59): Wikibase\Lib\SimpleCacheWithBagOStuff->get(string, string)
#2 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/Store/CachingFallbackLabelDescriptionLookup.php(120): Wikibase\Lib\StatsdRecordingSimpleCache->get(string, string)
#3 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/Store/CachingFallbackLabelDescriptionLookup.php(106): Wikibase\Lib\Store\CachingFallbackLabelDescriptionLookup->getTerm(Wikibase\DataModel\Entity\ItemId, string, string)
#4 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/client/includes/Usage/UsageTrackingLanguageFallbackLabelDescriptionLookup.php(71): Wikibase\Lib\Store\CachingFallbackLabelDescriptionLookup->getLabel(Wikibase\DataModel\Entity\ItemId)
#5 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/client/includes/DataAccess/Scribunto/WikibaseLanguageDependentLuaBindings.php(60): Wikibase\Client\Usage\UsageTrackingLanguageFallbackLabelDescriptionLookup->getLabel(Wikibase\DataModel\Entity\ItemId)
#6 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/client/includes/DataAccess/Scribunto/Scribunto_LuaWikibaseLibrary.php(605): Wikibase\Client\DataAccess\Scribunto\WikibaseLanguageDependentLuaBindings->getLabel(string)
#7 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxCallback.php(26): Wikibase\Client\DataAccess\Scribunto\Scribunto_LuaWikibaseLibrary->getLabel(string)
#8 [internal function]: Scribunto_LuaSandboxCallback->__call(string, array)
#9 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxInterpreter.php(113): LuaSandboxFunction->call(LuaSandboxFunction)
#10 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Scribunto/includes/engines/LuaCommon/LuaEngine.php(291): Scribunto_LuaSandboxInterpreter->callFunction(LuaSandboxFunction, LuaSandboxFunction)
#11 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Scribunto/includes/engines/LuaCommon/LuaModule.php(69): Scribunto_LuaEngine->executeFunctionChunk(LuaSandboxFunction, PPTemplateFrame_Hash)
#12 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Scribunto/includes/common/Hooks.php(128): Scribunto_LuaModule->invoke(string, PPTemplateFrame_Hash)
#13 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/Parser.php(3596): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#14 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/Parser.php(3300): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#15 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/PPFrame_Hash.php(253): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#16 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/Parser.php(3478): PPFrame_Hash->expand(PPNode_Hash_Tree)
#17 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/PPFrame_Hash.php(253): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#18 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/PPTemplateFrame_Hash.php(89): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#19 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/Parser.php(3475): PPTemplateFrame_Hash->cachedExpand(string, PPNode_Hash_Tree)
#20 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/PPFrame_Hash.php(253): Parser->braceSubstitution(array, PPFrame_Hash)
#21 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/Parser.php(3141): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#22 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/Parser.php(1475): Parser->replaceVariables(string)
#23 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/Parser.php(599): Parser->internalParse(string)
#24 /srv/mediawiki/php-1.35.0-wmf.23/includes/content/WikitextContent.php(368): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#25 /srv/mediawiki/php-1.35.0-wmf.23/includes/content/AbstractContent.php(565): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#26 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RenderedRevision.php(267): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#27 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RenderedRevision.php(236): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#28 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#29 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#30 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#31 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RenderedRevision.php(198): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#32 /srv/mediawiki/php-1.35.0-wmf.23/includes/poolcounter/PoolWorkArticleView.php(196): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#33 /srv/mediawiki/php-1.35.0-wmf.23/includes/poolcounter/PoolCounterWork.php(125): PoolWorkArticleView->doWork()
#34 /srv/mediawiki/php-1.35.0-wmf.23/includes/page/Article.php(809): PoolCounterWork->execute()
#35 /srv/mediawiki/php-1.35.0-wmf.23/includes/page/CategoryPage.php(66): Article->view()
#36 /srv/mediawiki/php-1.35.0-wmf.23/includes/actions/ViewAction.php(63): CategoryPage->view()
#37 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(519): ViewAction->show()
#38 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(305): MediaWiki->performAction(CategoryTreeCategoryPage, Title)
#39 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(972): MediaWiki->performRequest()
#40 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(535): MediaWiki->main()
#41 /srv/mediawiki/php-1.35.0-wmf.23/index.php(47): MediaWiki->run()
#42 /srv/mediawiki/w/index.php(3): require(string)
#43 {main}

Event Timeline

brennen created this task.Mar 11 2020, 9:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2020, 9:33 PM
brennen moved this task from Backlog to Logs/Train on the User-brennen board.Mar 11 2020, 10:56 PM
Krinkle added a subscriber: Krinkle.

This is a fatal error from WikibaseClient code as seen on group1 with Commons. The stack trace suggests it is encountered while parsing wikitext, which suggests to me that it is quite likely this will also start breaking Wikipedia pages views and make some pages unsaveable and/or onviewable.

Krinkle triaged this task as Unbreak Now! priority.Mar 12 2020, 1:28 AM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMar 12 2020, 1:28 AM

This is just T247057 again, right?

This is a fatal error from WikibaseClient code as seen on group1 with Commons. The stack trace suggests it is encountered while parsing wikitext, which suggests to me that it is quite likely this will also start breaking Wikipedia pages views and make some pages unsaveable and/or onviewable.

Yes, it comes from the Lua side, per @Tarrow at T247057#5948302.

Yup, I don't think it should be a train blocker though. The normal pageviews can't get broken, it breaks when you try to see the page in an invalid language and also it happens virtually on commons because the language they request is bound to user's language (most wikis request data in their language).

Yes, I don't think this should be a blocker; clearly I should have done a better job of fixing this symptom earlier rather than going for the cause since it's making so many people worry

I've just merged https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/572259 which I think will stem most (maybe all?) of these errors. From my side I'd be happy for this to not block the train in either case but we could also backport this band-aid to reduce the logspam

Thanks - and yeah, let's backport.

Change 579302 had a related patch set uploaded (by Tarrow; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.23] Do not try to load label in unknown languages in Lua

https://gerrit.wikimedia.org/r/579302

Change 579302 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.23] Do not try to load label in unknown languages in Lua

https://gerrit.wikimedia.org/r/579302

Resolved?

Nope :( I reverted the backport because it doesn't fix the issue.

We have been fixing the issue on WikibaseLanguageIndependentLuaBindings but it's actually happening on WikibaseLanguageDependentLuaBindings and LanguageFallbackLabelDescriptionLookup has the wrong language code, I have been trying to fix it by adding list of term languages as a dependency but the class is being instantiated in 22 different places (some outside of Wikibase git repo) making it pretty hard to fix :/ Let me find an interim fix for now.

brennen lowered the priority of this task from Unbreak Now! to Medium.Mar 12 2020, 6:59 PM

Given pre-existing nature of this breakage and per conversation in RelEng, we'll go ahead and remove this as a train blocker. It'd still definitely be good to reduce the log noise if possible.

Krinkle removed a subscriber: Krinkle.Mar 13 2020, 3:48 PM

Change 579302 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.23] Do not try to load label in unknown languages in Lua

https://gerrit.wikimedia.org/r/579302

This looks like it would fix the issue but I'm still seeing the same error in prod:

[fatal] CacheInvalidArgumentException:
Cache key contains characters that are not allowed: `Q49092_1112215108_⧼lang⧽_label`

#0 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/SimpleCacheWithBagOStuff.php(69)
… #5 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/client/includes/DataAccess/Scribunto/WikibaseLanguageDependentLuaBindings.php(60)
… #7 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxCallback.php(26)
… #13 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/Parser.php(3596)
… #33 /srv/mediawiki/php-1.35.0-wmf.23/includes/poolcounter/PoolCounterWork.php(125)
… #37 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(519): ViewAction->show()
… #41 /srv/mediawiki/php-1.35.0-wmf.23/index.php(47): MediaWiki->run()

Change 579302 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.23] Do not try to load label in unknown languages in Lua

https://gerrit.wikimedia.org/r/579302

This looks like it would fix the issue but I'm still seeing the same error in prod:

[fatal] CacheInvalidArgumentException:
Cache key contains characters that are not allowed: `Q49092_1112215108_⧼lang⧽_label`

#0 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/SimpleCacheWithBagOStuff.php(69)
… #5 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Wikibase/client/includes/DataAccess/Scribunto/WikibaseLanguageDependentLuaBindings.php(60)
… #7 /srv/mediawiki/php-1.35.0-wmf.23/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxCallback.php(26)
… #13 /srv/mediawiki/php-1.35.0-wmf.23/includes/parser/Parser.php(3596)
… #33 /srv/mediawiki/php-1.35.0-wmf.23/includes/poolcounter/PoolCounterWork.php(125)
… #37 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(519): ViewAction->show()
… #41 /srv/mediawiki/php-1.35.0-wmf.23/index.php(47): MediaWiki->run()

See T247466#5965249

Krinkle lowered the priority of this task from Medium to Low.Mar 16 2020, 6:57 PM
hashar added a subscriber: hashar.Mar 18 2020, 2:38 PM

I would guess uselang should be validated before being passed down? LanguageConverter seems to have some logic to validate it.

https://commons.wikimedia.org/wiki/Category:Subcultures?uselang=%E2%A7%BCLang%E2%A7%BD

The uselang is a red herring. Yes it can be strange, but that is supported. It is possible that there is a bug and that this value should be discarded, but that is not the bug for this task.

Our BagOStuff class allows any character to be used in a cache key, no problem. It just needs to be encoded correcrty, which happens automaticaly if you call makeKey() on the correct cache object - the same object that will store the value. The problem is that Wikibase's SimpleCacheWithBagOStuff, unlike all other proxy classes, is not proxying the makeKey methods correctly.

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptMar 19 2020, 3:57 PM