Between 2019-02-28T14:40 and 2019-02-28T14:45, this exception shows up in production for
some load.php requests all HTTP requests on mw1272.
I've never seen this error before and it seems rather concerning this can happen.
Seems like the cache somehow disappeared on a pooled production app server. Or perhaps a new branch enabled without the localisation cache being created first (which is a fatal error, given we've configured MW to disallow computing/populating LCStore on-demand.)
Update: The time frame correlates with Server Admin Log entries:
- 14:41 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.19 (duration: 00m 53s).
- 14:46 hashar@deploy1001: scap sync-l10n completed (1.33.0-wmf.19) (duration: 03m 33s).
- 14:46 hashar: mw1272 had /srv/mediawiki/php-1.33.0-wmf.19/includes/cache/localisation/LocalisationCache.php:475) No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php.
- 14:47 hashar: mw1272 fixed by running "scap sync-l10n" from deploy host.
What led to this problem, and how can we prevent this in the future?