Page MenuHomePhabricator

Update MediaWiki localisation in Wikimedia wikis daily
Open, Needs TriagePublic


To ensure the best user experience to all our users, whatever language they speak, we need to minimise exposure to untranslated messages and maximise our ability to correct mistakes in translation. Satisfying users of all languages has highest priority in the Wikimedia mission ("people around the world", "globally").

In MediaWiki, for many years now, this has been achieved by a process known in the industry as continuous localisation, where i18n code and l10n strings iterate quickly in an immediate feedback loop between developers, translators and users. This means that developers develop good quality messages (both i18n and l10n) at the same time as they develop all their code; translators immediately receive those messages and submit their translations, proving either that they're translatable or that some i18n fix is needed; the two groups are in constact contact (here on Phabricator mostly) to surface any i18n issues, which can therefore be corrected when the cost is lowest i.e. at the same time as initial development occurs; users see the translations immediately, so that any possible improvements are spotted (given enough eyeballs, are bugs are shallow) and ideally executed on the spot (all our users are also editors and translators, or can be).

Thanks to Raymond's daily work, is able to guarantee the maximum service quality to all MediaWiki repositories, which receive up to date translations every day, i.e. typically within 24 hours from the time translators provide them. Import runs every few hours, this means we're able to guarantee that a message is translated on the live wikis within 40 hours from being added, and possibly as little as 4 hours.

The most obvious solution to the problem is to run the existing l10nupdate every day. Any part of the infrastructure which happened to exhibit problems with this will need to be fixed consequently. This regression has lasted already way too long for us to continue sustaining its enormous opportunity cost. Micro-optimisations can and should be performed with lower priority. We know that this is not some unsolved problem in computer science because we already fixed it for 10 years. The need for a daily run will be considered for any refactoring or code change in the deployment systems, as a non-negotiable premise just like the daily sunrise.

The current configuration regression (T164035) also places an undue burden on individual deployers who are forced to extracurricular duties to guarantee mission-critical service (T203297#4564753, other one-time deployments etc.), negating also any short-term benefits it hoped to bring (T158360#3127368).

Event Timeline

Nemo_bis triaged this task as High priority.Sep 7 2018, 6:05 AM
greg raised the priority of this task from High to Needs Triage.Sep 7 2018, 2:45 PM
greg added a subscriber: greg.

Given the open RFC another task presupposing an outcome should be "Needs Triage".

How is this task different from T158360?

That ticket was a vague request to undeploy LU that got retitled into a re-evaluation because no one wants to remove it, except no one seems to want to work on it either (both TechCom and RelEng tried to say "not mine" at different points). And I get the feeling that the existence of that ticket has demotivated people from making small steps to improve LU.

This ticket has a well documented rationale with a specific focus, and a well documented resolution (running l10nupdate daily). I think there's value in tracking and resolving it separately.

Given the open RFC another task presupposing an outcome should be "Needs Triage".

Given the recent developments around T203297: Manage translations for, I do think this is high priority.

I am very much interested in having working LU (or equivalent) and willing to work on it. But I looked through the RFC and lists maybe one concrete issues and some more that are quite vague. More importantly, it's not clear to me which requirements need to be met to have LU run daily – I consider that as a precondition.

Just to be clear: what is being asked here is to resume running l10nupdate on the weekend (which was turned off due to issues with l10nupdate breakage). To speak to Niklas' question: who is going to be the point person/team for when l10nupdate breaks over the weekend?

We don't do deploys on the weekend. And when we do we have a policy to limit the deploy to as small of a change as possible (eg: sync-file) and shy away from running full scaps. l10nupdate runs a full scap.

So, to move us forward: let's have someone/team volunteer to be steward of l10nupdate and then collaboratively work through a list of improvements/changes to it. This could result in l10nupdate running every day, or it may not, depending on the work and what we learn as we go. For the record, the Language Team is already listed as stewards on the Dev/Maintainers page. I assume that is true and that's who will work on the improvements (when it is feasible with their other work priorities, of course).