Page MenuHomePhabricator

No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php. in production when running populateSitesTable for aawikibooks with foreachwikiindblist
Closed, ResolvedPublic

Description

The script run can be seen below and seems rather odd. It seems to be reproducible always failing on the same wikis. However running the script against the wiki by itself succeeds.

Tagging Ops and Releng as I'm not sure who technically owns this script.

1addshore@deploy1001:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https
2
3-----------------------------------------------------------------
4aawiki
5-----------------------------------------------------------------
6
7
8aawiki: done.
9-----------------------------------------------------------------
10aawikibooks
11-----------------------------------------------------------------
12[9e5de6a021baf53f38af2d22] [no req] MWException from line 505 of /srv/mediawiki/php-1.35.0-wmf.34/includes/cache/localisation/LocalisationCache.php: No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php.
13Backtrace:
14#0 /srv/mediawiki/php-1.35.0-wmf.34/includes/cache/localisation/LocalisationCache.php(363): LocalisationCache->initLanguage(string)
15#1 /srv/mediawiki/php-1.35.0-wmf.34/includes/cache/localisation/LocalisationCache.php(372): LocalisationCache->loadItem(string, string)
16#2 /srv/mediawiki/php-1.35.0-wmf.34/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem(string, string)
17#3 /srv/mediawiki/php-1.35.0-wmf.34/includes/language/LanguageFallback.php(106): LocalisationCache->getItem(string, string)
18#4 /srv/mediawiki/php-1.35.0-wmf.34/includes/language/LanguageFactory.php(167): MediaWiki\Languages\LanguageFallback->getAll(string)
19#5 /srv/mediawiki/php-1.35.0-wmf.34/includes/language/LanguageFactory.php(114): MediaWiki\Languages\LanguageFactory->newFromCode(string)
20#6 /srv/mediawiki/php-1.35.0-wmf.34/includes/ServiceWiring.php(212): MediaWiki\Languages\LanguageFactory->getLanguage(string)
21#7 /srv/mediawiki/php-1.35.0-wmf.34/includes/libs/services/ServiceContainer.php(451): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
22#8 /srv/mediawiki/php-1.35.0-wmf.34/includes/libs/services/ServiceContainer.php(419): Wikimedia\Services\ServiceContainer->createService(string)
23#9 /srv/mediawiki/php-1.35.0-wmf.34/includes/MediaWikiServices.php(605): Wikimedia\Services\ServiceContainer->getService(string)
24#10 /srv/mediawiki/php-1.35.0-wmf.34/includes/Setup.php(693): MediaWiki\MediaWikiServices->getContentLanguage()
25#11 /srv/mediawiki/php-1.35.0-wmf.34/maintenance/doMaintenance.php(89): require_once(string)
26#12 /srv/mediawiki/php-1.35.0-wmf.34/extensions/Wikibase/lib/maintenance/populateSitesTable.php(124): require_once(string)
27#13 /srv/mediawiki/multiversion/MWScript.php(101): require_once(string)
28#14 {main}
29-----------------------------------------------------------------
30aawiktionary
31-----------------------------------------------------------------
32[4e7da88936d0421cc1f42097] [no req] MWException from line 505 of /srv/mediawiki/php-1.35.0-wmf.34/includes/cache/localisation/LocalisationCache.php: No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php.
33Backtrace:
34#0 /srv/mediawiki/php-1.35.0-wmf.34/includes/cache/localisation/LocalisationCache.php(363): LocalisationCache->initLanguage(string)
35#1 /srv/mediawiki/php-1.35.0-wmf.34/includes/cache/localisation/LocalisationCache.php(372): LocalisationCache->loadItem(string, string)
36#2 /srv/mediawiki/php-1.35.0-wmf.34/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem(string, string)
37#3 /srv/mediawiki/php-1.35.0-wmf.34/includes/language/LanguageFallback.php(106): LocalisationCache->getItem(string, string)
38#4 /srv/mediawiki/php-1.35.0-wmf.34/includes/language/LanguageFactory.php(167): MediaWiki\Languages\LanguageFallback->getAll(string)
39#5 /srv/mediawiki/php-1.35.0-wmf.34/includes/language/LanguageFactory.php(114): MediaWiki\Languages\LanguageFactory->newFromCode(string)
40#6 /srv/mediawiki/php-1.35.0-wmf.34/includes/ServiceWiring.php(212): MediaWiki\Languages\LanguageFactory->getLanguage(string)
41#7 /srv/mediawiki/php-1.35.0-wmf.34/includes/libs/services/ServiceContainer.php(451): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
42#8 /srv/mediawiki/php-1.35.0-wmf.34/includes/libs/services/ServiceContainer.php(419): Wikimedia\Services\ServiceContainer->createService(string)
43#9 /srv/mediawiki/php-1.35.0-wmf.34/includes/MediaWikiServices.php(605): Wikimedia\Services\ServiceContainer->getService(string)
44#10 /srv/mediawiki/php-1.35.0-wmf.34/includes/Setup.php(693): MediaWiki\MediaWikiServices->getContentLanguage()
45#11 /srv/mediawiki/php-1.35.0-wmf.34/maintenance/doMaintenance.php(89): require_once(string)
46#12 /srv/mediawiki/php-1.35.0-wmf.34/extensions/Wikibase/lib/maintenance/populateSitesTable.php(124): require_once(string)
47#13 /srv/mediawiki/multiversion/MWScript.php(101): require_once(string)
48#14 {main}
49-----------------------------------------------------------------
50abwiki
51-----------------------------------------------------------------
52abwiki: done.
53-----------------------------------------------------------------
54abwiktionary
55-----------------------------------------------------------------
56

Running the script alone seems to work just fine:

addshore@deploy1001:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki aawikibooks --force-protocol https
done.
addshore@deploy1001:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki aawikibooks --force-protocol https
done.
addshore@deploy1001:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki aawikibooks --force-protocol https
done.

Event Timeline

Addshore created this task.May 27 2020, 1:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 27 2020, 1:38 PM
Krinkle added a subscriber: Krinkle.

This is used by critical cron jobs. Tentatively blocking until we know it's not applicable or solved. Errors like this have taken the site down more than once :)

zeljkofilipin triaged this task as Unbreak Now! priority.May 27 2020, 4:11 PM
zeljkofilipin added a subscriber: zeljkofilipin.

Train blockers are UBN.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMay 27 2020, 4:11 PM
mmodell added a subscriber: mmodell.EditedMay 27 2020, 4:44 PM

I didn't notice any errors related to updating localization cache yesterday during the train for testwikis/group0.

We should try to reproduce on mwmaint or only deploy1001.

Try foreachwiki with eval to see if we can access l10n there

@Krinkle irl

mmodell added a comment.EditedMay 27 2020, 6:16 PM

foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https

This command works on mwmaint1002.eqiad.wmnet and I've got it running currently, I'll let it finish.

mmodell closed this task as Resolved.May 27 2020, 6:35 PM
mmodell claimed this task.

/srv/mediawiki/php-*/cache/l10n is empty on deploy1001. Generally, you shouldn't run maintenance scripts on deployment hosts.

Interestingly, the reason this works with mwscript and not foreachwikiindblist is that mwscript has the following check that isn't in foreachwikiindblist:

# Use the source version of the script if the source directory is present.
# This only matters if the source dir is shared or when run on the deployment server.
if [ -d "$MEDIAWIKI_STAGING_DIR" ]; then
        MEDIAWIKI_DEPLOYMENT_DIR_DIR_USE=$MEDIAWIKI_STAGING_DIR
else
        MEDIAWIKI_DEPLOYMENT_DIR_DIR_USE=$MEDIAWIKI_DEPLOYMENT_DIR
fi
Reedy added a subscriber: Reedy.May 27 2020, 7:10 PM

Why didn't it break on aawiki though?

/srv/mediawiki/php-*/cache/l10n is empty on deploy1001. Generally, you shouldn't run maintenance scripts on deployment hosts.

Ooof, that is indeed my bad for running this in the wrong place.
I had my eyes open, but apparently not enough.

Interestingly, the reason this works with mwscript and not foreachwikiindblist is that mwscript has the following check that isn't in foreachwikiindblist:

# Use the source version of the script if the source directory is present.
# This only matters if the source dir is shared or when run on the deployment server.
if [ -d "$MEDIAWIKI_STAGING_DIR" ]; then
        MEDIAWIKI_DEPLOYMENT_DIR_DIR_USE=$MEDIAWIKI_STAGING_DIR
else
        MEDIAWIKI_DEPLOYMENT_DIR_DIR_USE=$MEDIAWIKI_DEPLOYMENT_DIR
fi

Might we be able to do something similar for foreachwikiindblist ?
Or even just spout some sort of INFO message or error saying you maybe should be running it somewhere else?