Page MenuHomePhabricator

L10n cache is completely broken
Closed, ResolvedPublic

Description

Manual localisation cache rebuild is failing with:

Query: INSERT  INTO `bw_l10n_cache` ......
Error: 1062 Duplicate entry 'mr-messages:apihelp-query+allusers-example-y' for key 'PRIMARY' (127.0.0.1:3312)

Language codes seen: mr and tl. It shouldn't even be using the database!

While trying to access the wiki from web I see:

Warning: dba_open(/l10n_cache-en.cdb.tmp.459236192): failed to open stream: Permission denied in /www/dev.translatewiki.net/docroot/w/vendor/wikimedia/cdb/src/Writer/DBA.php on line 39
[20897a16ed55e7907b3edc15] /w/i.php?title=Special:ContentTranslation&page=Moshing&from=en&to=es&targettitle=Moshing&debug=true MWException from line 98 of /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LCStoreCDB.php: Unable to open CDB file for write "/l10n_cache-en.cdb"

Backtrace:

#0 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(1015): LCStoreCDB->startWrite(string)
#1 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(457): LocalisationCache->recache(string)
#2 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(331): LocalisationCache->initLanguage(string)
#3 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(368): LocalisationCache->loadItem(string, string)
#4 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(293): LocalisationCache->loadSubitem(string, string, string)
#5 /www/dev.translatewiki.net/docroot/w/includes/cache/localisation/LocalisationCache.php(313): LocalisationCache->getSubitem(string, string, string)
#6 /www/dev.translatewiki.net/docroot/w/languages/Language.php(4485): LocalisationCache->getSubitemList(string, string)
#7 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(247): Language::getMessageKeysFor(string)
#8 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(1019): MessageCache->load(string)
#9 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(946): MessageCache->getMsgFromNamespace(string, string)
#10 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(916): MessageCache->getMessageForLang(LanguageEn, string, boolean, array)
#11 /www/dev.translatewiki.net/docroot/w/includes/cache/MessageCache.php(858): MessageCache->getMessageFromFallbackChain(LanguageEn, string, boolean)
#12 /www/dev.translatewiki.net/docroot/w/includes/language/Message.php(1310): MessageCache->get(string, boolean, LanguageEn)
#13 /www/dev.translatewiki.net/docroot/w/includes/language/Message.php(1029): Message->fetchMessage()
#14 /www/dev.translatewiki.net/docroot/w/extensions/CirrusSearch/includes/Hooks.php(198): Message->isDisabled()
#15 /www/dev.translatewiki.net/docroot/w/includes/libs/objectcache/BagOStuff.php(132): CirrusSearch\Hooks::CirrusSearch\{closure}(integer)
#16 /www/dev.translatewiki.net/docroot/w/extensions/CirrusSearch/includes/Hooks.php(202): BagOStuff->getWithSetCallback(string, integer, Closure)
#17 /www/dev.translatewiki.net/docroot/w/extensions/CirrusSearch/includes/Hooks.php(100): CirrusSearch\Hooks::overrideMoreLikeThisOptionsFromMessage()
#18 /www/dev.translatewiki.net/docroot/w/extensions/CirrusSearch/includes/Hooks.php(65): CirrusSearch\Hooks::initializeForRequest(WebRequest)
#19 /www/dev.translatewiki.net/docroot/w/includes/Hooks.php(174): CirrusSearch\Hooks::onBeforeInitialize(Title, NULL, OutputPage, User, WebRequest, MediaWiki)
#20 /www/dev.translatewiki.net/docroot/w/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#21 /www/dev.translatewiki.net/docroot/w/includes/MediaWiki.php(186): Hooks::run(string, array)
#22 /www/dev.translatewiki.net/docroot/w/includes/MediaWiki.php(892): MediaWiki->performRequest()
#23 /www/dev.translatewiki.net/docroot/w/includes/MediaWiki.php(523): MediaWiki->main()
#24 /www/dev.translatewiki.net/docroot/w/index.php(42): MediaWiki->run()
#25 {main}

It's trying to read CDB files from root!

My config:

$wgLocalisationCacheConf['store'] = 'files';
$wgLocalisationCacheConf['manualRecache'] = false;

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2019, 8:20 AM
Nikerabbit triaged this task as Unbreak Now! priority.Aug 26 2019, 8:30 AM
Nikerabbit added subscribers: Simetrical, daniel.

Git bisect points to rMWe4468a1d6b6b: Make LocalisationCache a service. Setting UBN! for attention. I feel this doesn't affect everyone, though, or it would have been noticed already.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptAug 26 2019, 8:30 AM

Does not revert cleanly. Only workaround seems to be to roll back to commit before it.

You need to revert LanguageNameUtils too, then it works fine (other than RELEASE-NOTES conflicts, obviously): 2e52f48c2ed

Obviously someone should revert immediately. Then I want to figure out what's going on with the failures -- this patch was from before I tried to get 100% test coverage before refactoring anything.

I think I see the problem:

+        $storeArg['directory'] =
+            $conf['storeDirectory'] ?? $services->getMainConfig()->get( 'CacheDirectory' );

vs.

-        $directory = !empty( $conf['storeDirectory'] ) ? $conf['storeDirectory'] : $wgCacheDirectory;

That needs to be ?:, not ??. I'll post a fix in a second.

Change 532343 had a related patch set uploaded (by simetrical; owner: simetrical):
[mediawiki/core@master] Correctly register storeDirectory in l10n cache

https://gerrit.wikimedia.org/r/532343

The breakage of rebuildLocalisationCache.php is an entirely separate problem. Separate patch coming for that. See, this is why tests before refactoring is a good idea!

So those two patches should fix the two issues that you ran into. I leave it up to the people with +2 access to decide whether it's wise to just commit these two fixes or to revert LocalisationCache and LanguageNameUtils for now.

Change 532346 had a related patch set uploaded (by simetrical; owner: simetrical):
[mediawiki/core@master] Pass correct store to rebuildLocalisationCache.php

https://gerrit.wikimedia.org/r/532346

Change 532343 merged by jenkins-bot:
[mediawiki/core@master] Correctly register storeDirectory in l10n cache

https://gerrit.wikimedia.org/r/532343

Change 532346 merged by jenkins-bot:
[mediawiki/core@master] Pass correct store to rebuildLocalisationCache.php

https://gerrit.wikimedia.org/r/532346

This should be fixed now, can you test?

(Not related or affecting translatewiki.net at the moment).

No directly observable errors seen now.

Then I think it's resolved, although the patch is being backed out anyway for now because of T231198/T231200.

Friendly reminder, this is blocking the train. Will it be resolved by train window tomorrow?

Tarrow added a subscriber: Tarrow.Aug 26 2019, 7:43 PM
Simetrical closed this task as Resolved.Aug 27 2019, 7:15 AM
Simetrical claimed this task.