Page MenuHomePhabricator
Create Task
Maniphest T231183

L10n cache is completely broken
Closed, ResolvedPublic
Actions

Description

Manual localisation cache rebuild is failing with:

Query: INSERT  INTO `bw_l10n_cache` ......
Error: 1062 Duplicate entry 'mr-messages:apihelp-query+allusers-example-y' for key 'PRIMARY' (127.0.0.1:3312)

Language codes seen: mr and tl. It shouldn't even be using the database!

While trying to access the wiki from web I see:

Warning: dba_open(/l10n_cache-en.cdb.tmp.459236192): failed to open stream: Permission denied in /www/dev.translatewiki.net/docroot/w/vendor/wikimedia/cdb/src/Writer/DBA.php on line 39
[20897a16ed55e7907b3edc15] /w/i.php?title=Special:ContentTranslation&page=Moshing&from=en&to=es&targettitle=Moshing&debug=true MWException from line 98 of /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LCStoreCDB.php: Unable to open CDB file for write "/l10n_cache-en.cdb"

Backtrace:

#0 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(1015): LCStoreCDB->startWrite(string)
#1 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(457): LocalisationCache->recache(string)
#2 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(331): LocalisationCache->initLanguage(string)
#3 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(368): LocalisationCache->loadItem(string, string)
#4 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(293): LocalisationCache->loadSubitem(string, string, string)
#5 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(313): LocalisationCache->getSubitem(string, string, string)
#6 /www/dev.translatewiki.net/docroot/w/languages/Language.php(4485): LocalisationCache->getSubitemList(string, string)
#7 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(247): Language::getMessageKeysFor(string)
#8 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(1019): MessageCache->load(string)
#9 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(946): MessageCache->getMsgFromNamespace(string, string)
#10 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(916): MessageCache->getMessageForLang(LanguageEn, string, boolean, array)
#11 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(858): MessageCache->getMessageFromFallbackChain(LanguageEn, string, boolean)
#12 /www/dev.translatewiki.net/docroot/w/includes/language/Message.php(1310): MessageCache->get(string, boolean, LanguageEn)
#13 /www/dev.translatewiki.net/docroot/w/includes/language/Message.php(1029): Message->fetchMessage()
#14 /www/dev.translatewiki.net/docroot/w/extensions/CirrusSearch/includes/Hooks.php(198): Message->isDisabled()
#15 /www/dev.translatewiki.net/docroot/w/includes/libs/objectcache/BagOStuff.php(132): CirrusSearch\Hooks::CirrusSearch\{closure}(integer)
#16 /www/dev.translatewiki.net/docroot/w/extensions/CirrusSearch/includes/Hooks.php(202): BagOStuff->getWithSetCallback(string, integer, Closure)
#17 /www/dev.translatewiki.net/docroot/w/extensions/CirrusSearch/includes/Hooks.php(100): CirrusSearch\Hooks::overrideMoreLikeThisOptionsFromMessage()
#18 /www/dev.translatewiki.net/docroot/w/extensions/CirrusSearch/includes/Hooks.php(65): CirrusSearch\Hooks::initializeForRequest(WebRequest)
#19 /www/dev.translatewiki.net/docroot/w/includes/Hooks.php(174): CirrusSearch\Hooks::onBeforeInitialize(Title, NULL, OutputPage, User, WebRequest, MediaWiki)
#20 /www/dev.translatewiki.net/docroot/w/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#21 /www/dev.translatewiki.net/docroot/w/includes/MediaWiki.php(186): Hooks::run(string, array)
#22 /www/dev.translatewiki.net/docroot/w/includes/MediaWiki.php(892): MediaWiki->performRequest()
#23 /www/dev.translatewiki.net/docroot/w/includes/MediaWiki.php(523): MediaWiki->main()
#24 /www/dev.translatewiki.net/docroot/w/index.php(42): MediaWiki->run()
#25 {main}

It's trying to read CDB files from root!

My config:

$wgLocalisationCacheConf['store'] = 'files';
$wgLocalisationCacheConf['manualRecache'] = false;

Details

SubjectRepoBranchLines +/-
Pass correct store to rebuildLocalisationCache.phpmediawiki/coremaster+38 -25
Correctly register storeDirectory in l10n cachemediawiki/coremaster+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedReleasezeljkofilipinT220745 1.34.0-wmf.20 deployment blockers
 ResolvedSimetricalT231183 L10n cache is completely broken

Event Timeline

Nikerabbit created this task.Aug 26 2019, 8:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2019, 8:20 AM
Nikerabbit triaged this task as Unbreak Now! priority.Aug 26 2019, 8:30 AM
Nikerabbit added subscribers: Simetrical, daniel.

Git bisect points to rMWe4468a1d6b6b: Make LocalisationCache a service. Setting UBN! for attention. I feel this doesn't affect everyone, though, or it would have been noticed already.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptAug 26 2019, 8:30 AM
Nikerabbit added a comment.Aug 26 2019, 8:52 AM

Does not revert cleanly. Only workaround seems to be to roll back to commit before it.

Simetrical added a comment.Aug 26 2019, 9:34 AM

You need to revert LanguageNameUtils too, then it works fine (other than RELEASE-NOTES conflicts, obviously): 2e52f48c2ed

Obviously someone should revert immediately. Then I want to figure out what's going on with the failures -- this patch was from before I tried to get 100% test coverage before refactoring anything.

Simetrical added a comment.Aug 26 2019, 9:41 AM

I think I see the problem:

+        $storeArg['directory'] =
+            $conf['storeDirectory'] ?? $services->getMainConfig()->get( 'CacheDirectory' );

vs.

-        $directory = !empty( $conf['storeDirectory'] ) ? $conf['storeDirectory'] : $wgCacheDirectory;

That needs to be ?:, not ??. I'll post a fix in a second.

gerritbot added a comment.Aug 26 2019, 9:45 AM

Change 532343 had a related patch set uploaded (by simetrical; owner: simetrical):
[mediawiki/core@master] Correctly register storeDirectory in l10n cache

https://gerrit.wikimedia.org/r/532343

gerritbot added a project: Patch-For-Review.Aug 26 2019, 9:45 AM
Simetrical added a comment.Aug 26 2019, 9:46 AM

The breakage of rebuildLocalisationCache.php is an entirely separate problem. Separate patch coming for that. See, this is why tests before refactoring is a good idea!

Simetrical added a comment.Aug 26 2019, 9:56 AM

So those two patches should fix the two issues that you ran into. I leave it up to the people with +2 access to decide whether it's wise to just commit these two fixes or to revert LocalisationCache and LanguageNameUtils for now.

gerritbot added a comment.Aug 26 2019, 9:57 AM

Change 532346 had a related patch set uploaded (by simetrical; owner: simetrical):
[mediawiki/core@master] Pass correct store to rebuildLocalisationCache.php

https://gerrit.wikimedia.org/r/532346

gerritbot added a comment.Aug 26 2019, 1:54 PM

Change 532343 merged by jenkins-bot:
[mediawiki/core@master] Correctly register storeDirectory in l10n cache

https://gerrit.wikimedia.org/r/532343

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.20; 2019-08-27).Aug 26 2019, 2:00 PM
gerritbot added a comment.Aug 26 2019, 3:06 PM

Change 532346 merged by jenkins-bot:
[mediawiki/core@master] Pass correct store to rebuildLocalisationCache.php

https://gerrit.wikimedia.org/r/532346

Maintenance_bot removed a project: Patch-For-Review.Aug 26 2019, 3:10 PM
Simetrical added a comment.Aug 26 2019, 4:02 PM

This should be fixed now, can you test?

Krinkle added a project: translatewiki.net.Aug 26 2019, 4:09 PM
Krinkle added a project: Platform Engineering.
Lucas_Werkmeister_WMDE mentioned this in T231198: Wikibase and WikibaseLexeme tests now take more than 1 hour.Aug 26 2019, 4:36 PM
Jdforrester-WMF added a parent task: T220745: 1.34.0-wmf.20 deployment blockers.Aug 26 2019, 4:51 PM
WMDE-leszek subscribed.Aug 26 2019, 4:52 PM
Nikerabbit removed a project: translatewiki.net.Aug 26 2019, 5:38 PM

(Not related or affecting translatewiki.net at the moment).

No directly observable errors seen now.

Simetrical added a comment.Aug 26 2019, 5:51 PM

Then I think it's resolved, although the patch is being backed out anyway for now because of T231198/T231200.

zeljkofilipin subscribed.Aug 26 2019, 6:02 PM

Friendly reminder, this is blocking the train. Will it be resolved by train window tomorrow?

Ladsgroup subscribed.Aug 26 2019, 6:11 PM

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/532399 resolves this as well.

WDoranWMF moved this task from Inbox to Tracking/Watching on the Platform Engineering board.Aug 26 2019, 6:12 PM
Tarrow subscribed.Aug 26 2019, 7:43 PM
Simetrical closed this task as Resolved.Aug 27 2019, 7:15 AM
Simetrical claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL