Is this causing the many TypeError fatal exceptions currently occurring on MediaWiki.org?

The error seems to go away after purging the page's cache.

Essentially same as T415619: Creation of dynamic property MediaWiki\Language\Dependency\FileDependency::$filename is deprecated {"exception":"[object] (ErrorException(code: 0) but this one is for WMF production. Translate extension is just a casualty here.

What are our options here? Is this just specifically to the Translate extension or is it hitting other code paths? Can we bump Translate's cache epoch somehow?

Making a direct blocker of the train, given that it's still an issue.

For Translate, running createMessageIndex.php should regenerate caches. I am not sure if anything else is impacted.

Mentioned in SAL (#wikimedia-operations) [2026-01-28T13:33:05Z] <jforrester@deploy2002> mwscript-k8s job started: /srv/mediawiki/php-1.46.0-wmf.13/extensions/Translate/scripts/createMessageIndex.php --wiki=mediawikiwiki # T415725

Not sure if that will fix it, but let's give it a go.

These are now also showing up at a lower roughly similar volume for testwiki on .13.

These are now also showing up at a lower roughly similar volume for testwiki on .13.

...correction, mostly for test.wikidata.org, I think?

For Translate, running createMessageIndex.php should regenerate caches. I am not sure if anything else is impacted.

Not sure if that will fix it, but let's give it a go.

I gather this didn't work. Do we have other options?

In T415725#11561785, @Nikerabbit wrote:

Essentially same as T415619: Creation of dynamic property MediaWiki\Language\Dependency\FileDependency::$filename is deprecated {"exception":"[object] (ErrorException(code: 0) but this one is for WMF production. Translate extension is just a casualty here.

That task includes a change which bumped a cache version in one file (FileBasedMessageGroupFactory.php), but there are five four other files with a getCacheVersion() function; is it worth bumping those as well? (Several of them sound like they could be related to the MessageGroupProcessing stuff in the stack trace.) [Edit: one of the files below is an interface.]

$ git grep 'function getCacheVersion'
src/MessageBundleTranslation/MessageBundleMessageGroupFactory.php:      public function getCacheVersion(): int {
src/MessageGroupConfiguration/FileBasedMessageGroupFactory.php: public function getCacheVersion(): int {
src/MessageGroupConfiguration/HookDefinedMessageGroupFactory.php:       public function getCacheVersion(): int {
src/MessageGroupProcessing/AggregateGroupMessageGroupFactory.php:       public function getCacheVersion(): int {
src/MessageGroupProcessing/CachedMessageGroupFactory.php:       public function getCacheVersion(): int;
src/PageTranslation/TranslatablePageMessageGroupFactory.php:    public function getCacheVersion(): int {

(I’ll also point out that James only ran createMessageIndex.php on mediawikiwiki for now, so if the errors had shifted to testwikidatawiki, that might indicate that the maintenance script had actually helped. But I don’t think that’s the case, AFAICT Logstash doesn’t show any change in the error rate on mediawikiwiki.)

Change #1234450 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] Update cache version for message group caches

https://gerrit.wikimedia.org/r/1234450

Change #1234451 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@wmf/1.46.0-wmf.13] Update cache version for message group caches

https://gerrit.wikimedia.org/r/1234451

Change #1234451 merged by jenkins-bot:

[mediawiki/extensions/Translate@wmf/1.46.0-wmf.13] Update cache version for message group caches

https://gerrit.wikimedia.org/r/1234451

Mentioned in SAL (#wikimedia-operations) [2026-01-28T17:15:40Z] <brennen@deploy2002> Started scap sync-world: Backport for [[gerrit:1234451|Update cache version for message group caches (T415725)]]

Mentioned in SAL (#wikimedia-operations) [2026-01-28T17:17:58Z] <brennen@deploy2002> abi, brennen: Backport for [[gerrit:1234451|Update cache version for message group caches (T415725)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-01-28T17:25:51Z] <brennen@deploy2002> Finished scap sync-world: Backport for [[gerrit:1234451|Update cache version for message group caches (T415725)]] (duration: 10m 11s)

In T415725#11562715, @Lucas_Werkmeister_WMDE wrote:

That task includes a change which bumped a cache version in one file (FileBasedMessageGroupFactory.php), but there are five four other files with a getCacheVersion() function; is it worth bumping those as well?

Done with the above backport but doesn’t seem to have helped :(

Ok, writing out the train-deployer-asks-stupid-questions I currently have about this.

First, I note that the overwhelming majority of errors seen here are for 1.46.0-wmf.12, on mw.org, which was briefly on group0 and then rolled back. The rest look to me like they're on test.wikidata.org (most) and test.wikipedia.org (a few), currently on .13. Everything seems to be for Translate.

Is our model of this:

1. Cache state still becomes broken under the current version of the code, or
2. Cache state becomes broken as a product of a rollback, or
3. There's just a bunch of lingering bad state from the now-fixed T415619

If it's #3, should this actually block the train? Understanding that the elevated error level is itself an ongoing problem, will the error rate from this problem change if the train rolls forward or is that an unknown?

I've seen it claimed that ?action=purge for a given page fixes the error, but @Lucas_Werkmeister_WMDE says not always, so maybe people are seeing a coincidence with something dropping out of cache?

Is there a CACHE_VERSION that we can bump for the LocalisationCache or is that not possible or not fix?

locally, rebuilding the localisation cache fixed issues for me but in prod, idk if that works perfectly.

In T415725#11563488, @brennen wrote:

3. There's just a bunch of lingering bad state from the now-fixed T415619

If it's #3, should this actually block the train? Understanding that the elevated error level is itself an ongoing problem, will the error rate from this problem change if the train rolls forward or is that an unknown?

I think it's #3, but I'm not sure.

In T415725#11563534, @aude wrote:

Is there a CACHE_VERSION that we can bump for the LocalisationCache or is that not possible or not fix?

I don't know if that would fix things; includes/Language/MessageCache.php is pretty far away from the breakage. We could try it. Obviously it'll be a slow deploy, but I don't have other ideas.

locally, rebuilding the localisation cache fixed issues for me but in prod, idk if that works perfectly.

In that case, this sounds like it's worth doing.

In T415725#11563488, @brennen wrote:

I've seen it claimed that ?action=purge for a given page fixes the error, but @Lucas_Werkmeister_WMDE says not always, so maybe people are seeing a coincidence with something dropping out of cache?

Yes, it (or null-edit) helps only once for a page and after revisit (same user or another) the error returns.

I'm catching up.

Here's what I think I understand and questions:

• Root problem is now fixed
• There's still a problem with _something_ that's cached
  • Something to do with l10n.
  • what cache are we talking about?
  • Is there a disagreement between something in ...some... cache (the messageBlobStore?) and the l10n cdb files on disk?
• Locally, rebuilding l10n fixes it
  • If we didn't move train, and rebuilt the l10n cache—is the theory that this problem goes away?
  • How can we verify this without making things worse?

In T415725#11563534, @aude wrote:

Is there a CACHE_VERSION that we can bump for the LocalisationCache or is that not possible or not fix?

locally, rebuilding the localisation cache fixed issues for me but in prod, idk if that works perfectly.

Per a chat with @dancy :

Hmm. Looks like there's no simple interface to this but one way would be to delete the l10n_cache-en.cdb file.

So we have a way of doing that, is that the right thing to do?

dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1088

sync-world: Add --force-l10n-update flag

Cross-posting from IRC:

[16:01:15]  <thcipriani> James_F: I still haven't actually caught one of these things, the pages seem to get fixed(?) before I can check them out.
[16:02:23]  <James_F> thcipriani: I get one right now at e.g. https://www.mediawiki.org/wiki/MediaWiki_1.39/fr
[16:03:22]  <thcipriani> :/ no error on the page as a user for me, logged in or logged out
[16:03:37]  <brennen> yeah, same.  this is just generally weird.
[16:03:48]  <James_F> Maybe it's DC-related somehow?
[16:03:57]  <James_F> Which'd be even worse to debug.
[16:04:41]  <James_F> Aha.
[16:05:09]  <James_F> if I use k8s-mwdebug-eqiad it fatals. If I use k8s-mwdebug-codfw it renders correctly.
[16:05:17]  <James_F> Have we somehow got a cache split?
[16:05:24]  <thcipriani> oh goodie
[16:05:27] brennen raises eyebrow
[16:05:34]  <James_F> If so, that's… worse.
[16:06:34]  <James_F> Same differential (works on codfw, dies on eqiad) for k8s-mwdebug-next-* and k8s-mwdebug-experimental-* for me.
[16:06:47]  <James_F> This feels like it's a clue.

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1088

sync-world: Add --force-l10n-update flag

Not sure if it helps, but if you're looking for examples, I still get a fatal on https://www.mediawiki.org/wiki/Skin:Vector whether logged in or logged out.

Another example: https://www.mediawiki.org/wiki/Extension:SyntaxHighlight

Mentioned in SAL (#wikimedia-operations) [2026-01-28T21:34:01Z] <brennen@deploy2002> Started scap sync-world: Syncing with --force-l10n-update to see if it clears out T415725

In T415725#11563955, @thcipriani wrote:

In T415725#11563534, @aude wrote:

Is there a CACHE_VERSION that we can bump for the LocalisationCache or is that not possible or not fix?

locally, rebuilding the localisation cache fixed issues for me but in prod, idk if that works perfectly.

Per a chat with @dancy :

Hmm. Looks like there's no simple interface to this but one way would be to delete the l10n_cache-en.cdb file.

So we have a way of doing that, is that the right thing to do?

We'll try this. The discussion with James_F (i.e., that this works differently in different datacenters that should already have identical l10n files) makes me think this might not work, but it seems safe to try—maybe there's a secondary effect here that I don't grok.

Mentioned in SAL (#wikimedia-operations) [2026-01-28T22:10:09Z] <brennen@deploy2002> Finished scap sync-world: Syncing with --force-l10n-update to see if it clears out T415725 (duration: 36m 46s)

Unfortunately, scap sync-world --force-l10n-update had no obvious effect.

are the l10n_cache-en.cdb files shared across versions of mediawiki in the cluster?

or does each version have it's own set of files?

In T415725#11563856, @Wargo wrote:

In T415725#11563488, @brennen wrote:

I've seen it claimed that ?action=purge for a given page fixes the error, but @Lucas_Werkmeister_WMDE says not always, so maybe people are seeing a coincidence with something dropping out of cache?

Yes, it (or null-edit) helps only once for a page and after revisit (same user or another) the error returns.

I don't know how short lived this workaround is but one the pages I purged earlier worked for awhile not just once. Now it showed the same error again and I just purged it again and confirmed that it works even when visiting from another device.

In T415725#11564233, @aude wrote:

are the l10n_cache-en.cdb files shared across versions of mediawiki in the cluster?

or does each version have it's own set of files?

Each version has it's own set of files, as it has for many many years :)

In an as-yet-unexplained update: the error we were seeing stopped at 22:42:30-ish

Yes, random translated/translatable pages are OK. Tested on mediawikiwiki and testwiki.

Copying some things I said in a Slack thread (yeah, I know) here for more visibility:

__PHP_Incomplete_Class is what unserialize() returns when the PHP class specified by the serialization data is not found in the current scope. My sense making is that there are copies of the namespaced MediaWiki\Language\Dependency\* classes in the WANCache (memcached behind mcrouter) for some keys and now that the namespaced versions of those classes have been reverted when the keys are hit unserialize says LOL I have no idea what you wanted, here have a placeholder object.

Adding some biz logic in WANObjectCache to check for __PHP_Incomplete_Class returns from unserialize and log what the __PHP_Incomplete_Class_Name and key are would maybe be nice.

Treating an __PHP_Incomplete_Class result as a cache miss would likely work too.

@matmarex mentioned that T389402: Consider adding an unserialize_callback_func for MediaWiki to handle errors from unserialisation was an idea extracted from T388725: PHP Warning: Class __PHP_Incomplete_Class has no unserializer which was a very similar issue 10 months ago.

Just as a side-note/follow-up note re comms: in hindsight (& IMHO), this might've been a good thing to have had an entry on https://wikimediastatus.net/ about; given the noticeable end-user impact it had on MW.org

In T415725#11564347, @brennen wrote:

Leaving this open in case of followups with cache mechanisms, removing as train blocker.

Re follow-up actions with e.g. cache mechanisms, IMO it might be good to split these follow-ups into separate tasks (& resolve this one), to e.g. prevent potential confusion from a task about this specific error still being open some time after the error itself has been resolved (& to make clearer at a glance what the proposed follow-up actions are).

I've not seen any stack trace that points to l10n cache. Only Translate's message group cache. Either version of codebase should work fine, but not when migrating one version to another or two codebases sharing the same cache value. In addition, there might be multiple cache values, one for each datacenter.

In theory createMessageIndex.php should work cause cache values to be regenerated in all datacenters. Likely the failure to load the value broke the regeneration though, and the fix would be to run the script for each wiki on each datacenter.

Disclaimer: I have not verified any of this, just my reflections based on the comments here and what we did for translatewiki.net.

Change #1234450 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Update cache version for message group caches

https://gerrit.wikimedia.org/r/1234450