Page MenuHomePhabricator

Add ability to ignore missing extensions in mergeMessageFileList's `--list-file` input
Open, Needs TriagePublicFeature

Description

As a Wikimedia depolyer
I want to use a unified extensions list with mergeMessageFileList across multiple deployments and production & testing environments
Without needing the exact set of extensions to be deployed in all deployments
So I can keep configuration management toil low while also avoiding early deployment of code and arbitrary deployment delays.

This is a very Wikimedia Beta Cluster and production deployment train coded desire and problem. In rOMWCaa11a2c0519e: Moving PerformanceInspector to extension-list a configuration file was removed that previously tracked extensions in Beta Cluster which were not yet deployed to production. A side effect of this configuration consolidation is that to be enabled in Beta Cluster an extension must be added to wmf-config/extension-list. This was explained at the time as an intermediate step towards removing the need for using mergeMessageFileList --list-file entirely. That follow up work however has not yet happened 7 years later. Instead today we not only couple the Beta Cluster and production extension-list to each other, we also couple the extension-list for all active productions branches. The net result is that before being enabled in Beta Cluster an extension's PHP code must be deployed without being activated in at least two consecutive deployment trains. Only after all active production branch deployments contain a given extension or skin can it then be safely added to wmf-config/extension-list for building localization cache in Beta Cluster and production.

Chad's original vision of replacing mergeMessageFileList --list-file with mergeMessageFileList --extension-dir in Wikimedia deployments is still possible, but it will require some changes to Beta Cluster deployments that we do not have a good plan for yet. If we loosen the constraint that all extensions and skins named in the mergeMessageFileList --list-file input must be present we can remove the necessity of all production and Beta Cluster deployments having the same files available locally via a slightly different path.

I propose that we add an --allow-missing-extensions cli option to mergeMessageFileList` which when enabled will log a warning about any missing extension or skin rather than raising a fatal error. Once this functionality is available across Beta Cluster and all active WMF production MediaWiki branches we can change scap to use it and break the deployed file coupling.

Event Timeline

I have recently been talking with @Catrope and @thcipriani about the underlying problem here and would like their feedback on my problem description and proposed solution. Obviously input from others is welcome as well.

I turns out that the current mergeMessageFileList.php implementation does not treat missing extensions in the list-file input as a fatal error. It does just what I proposed: logs the missing extension/skin and moves on. There was a hard failure when rMW8da7f5eeff33: Added $wgExtensionEntryPointListFiles for use with mergeMessageFileList.php was first implemented, but that was removed by rMW76680021f80e: Don't hard fail when we couldn't find an entry point for an extension way back in 2016.

So I guess there is nothing to do here on the MediaWiki side or even the scap side? We just need to change the process of how we use wmf-config/extension-list and start adding extensions & skins there when they are wanted in Beta Cluster while ignoring that this will temporarily generate warnings in the mergeMessageFileList.php runtime output.

Is there another reason that we have the local lore that any extension or skin must be in all deployed branches before being added to wmf-config/extension-list?

Is there another reason that we have the local lore that any extension or skin must be in all deployed branches before being added to wmf-config/extension-list?

hrm, my memory here is that generating the message file list wasn't the problem, but the step beyond that. Maybe something to do with rebuildLocalisationCache.php.

Anyway, probably fastest to try it out and see what happens!

I have recently been talking with @Catrope and @thcipriani about the underlying problem here and would like their feedback on my problem description and proposed solution. Obviously input from others is welcome as well.

Your problem description sounds right to me, and your proposed solution sounds good. My only concern is that I'd still want there to be a way to detect typos, broken references or missing extensions in extension-list, so that we don't accidentally make an entire extension's i18n disappear or deploy a new extension without ever adding it to extension-list. But that can probably be done in other ways, like:

  • A CI job in the mediawiki-config repo that verifies that every entry in extension-list has a corresponding wfLoadExtension call in CommonSettings.php and vice versa
  • A warning when an entry in extension-list points to a missing extension. If that's too spammy to do for every scap run, maybe we could do it more prominently for deployment trains, or have the automated testwiki deploy job send a warning message somewhere if it detects a missing extension.

hrm, my memory here is that generating the message file list wasn't the problem, but the step beyond that. Maybe something to do with rebuildLocalisationCache.php.

My memory is the opposite (that generating the message file list was where the error was), but all the more reason to try it and see what happens, as Tyler said.

Hmm now that I'm thinking about it a little more... do we run mergeMessageFileList for each wmf.N branch separately? If we run it only once and reuse the result across both branches, that would be a problem if an extension was missing in one branch but present in the other.

Hmm now that I'm thinking about it a little more... do we run mergeMessageFileList for each wmf.N branch separately? If we run it only once and reuse the result across both branches, that would be a problem if an extension was missing in one branch but present in the other.

Scap processes l10n for each wmf.N branch separately.

Tried adding a non-existent extension to extension-list in train-dev: went fine...maybe this has been a documentation problem for years?