Page MenuHomePhabricator

Allow delivery of multilingual content (TechNews, Editing newsletter, ...) in fallback languages
Open, MediumPublic

Description

Please make it so that when such mass messages are sent, wikis can receive them in their fallback language if their main one is not available.
(Please also add other tags if relevant.)

QA plan

Affected projects: multilingual Wikimedia projects, mainly MetaWiki
Pre-deployment testing: local testing by devs
Post-deployment testing: try sending out a translatable page as a mass message, on MetaWiki, to different projects, on pages in different languages, with also testing fallbacks

Outcome

Added support for sending multilingual pages–using fallback languages if necessary–to the Mass Message extension

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2017, 7:37 AM

Or in user-selected languages (unless this is already happening?)

Elitre added a subscriber: Johan.May 31 2017, 3:34 PM

As my colleague @Johan says, It's even worse for e.g. the Spanish Village Pump on Commons, which will get [messages] in English.
We currently can't do any finer tuning than per wiki, when it comes to languages.

It should be easy for MassMessage to call Translate APIs to load the messages in another language. I guess the more complex part is figuring out the language.

Elitre added a comment.Jun 5 2017, 7:42 PM

Well, although burdensome, we could add two language attributes to https://meta.wikimedia.org/wiki/Template:Target .
It could look like {{target | user=Elitre | site = it.wikipedia.org| lang = it | fallback=nap}}, so it could work in the same way for users and community pages?
(don't yell at me if it's unfeasible, tell me who I have to bribe to make this happen :) )

Elitre added a subscriber: Amire80.Jul 10 2018, 4:43 PM

As my colleagues just reminded me: pt-br gets translated weekly, but pt doesn't, so pt wikis get Tech News in English. (cc @Amire80 who will probably recall this.)

Pginer-WMF triaged this task as Medium priority.Jan 20 2020, 1:37 PM
abi_ added a subscriber: abi_.Mar 1 2020, 6:22 AM

We've reviewed how the Tech news gets sent out on a weekly basis. Here's a a summary of steps,

  1. Prepare the tech news in English, and mark the page for translation.
  2. Send a message on translators-l list requesting translators to translate it into different languages.
  3. Translators submit translations via the Translate extension.
  4. Generate this message from all the translations using the Lua script here.
  5. Send the generated message using the MassMessage extension.

The message contains a switch case provided by the Parser function extension. So currently MassMessage does not care about the source language of the wiki, or the language of the message. It just takes the messages and pushes it to all the pages on given wiki. Thanks to @Johan for his inputs.

There are two possible ways to handle this,

  1. Update the Lua script or add another parser function to handle fallback languages.
  2. Allow MassMessage extension to take a translatable page as input for the message to be sent.

We're currently planning to implement option 2. This has the following benefits,

  1. Simpler process for sending out the tech news, where step 4 above will no longer be needed.
  2. Address or work towards addressing other issues such as,

Fallback languages can be determined by using the LanguageFallback class,

$service = MediaWikiServices::getInstance();
// => MediaWiki\MediaWikiServices {#15}
$service->getLanguageFallback();
// => MediaWiki\Languages\LanguageFallback {#139}
$lang = $service->getLanguageFallback();
// => MediaWiki\Languages\LanguageFallback {#139}
$lang->getAll( 'pt-br' )

Change 576850 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] [WIP] Add a message page field on the Special:MassMessage page

https://gerrit.wikimedia.org/r/576850

Change 578467 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Update jobs to process page-message parameter

https://gerrit.wikimedia.org/r/578467

Testing is blocked on actually getting the patches merged: T249194: Deprecated: Use of parserTest v1 in Scribunto and SyntaxHighlight
I gave myself necessary rights on the beta cluster and started https://meta.wikimedia.beta.wmflabs.org/wiki/T165128_testing_list

Change 576850 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Add a page selector on Special:MassMessage to be sent as message

https://gerrit.wikimedia.org/r/576850

Change 578467 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Update jobs to process the messages that contain a page

https://gerrit.wikimedia.org/r/578467

Change 586353 had a related patch set uploaded (by Nikerabbit; owner: Nikerabbit):
[operations/mediawiki-config@master] Restore Beta Cluster logging

https://gerrit.wikimedia.org/r/586353

Change 586357 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Use Status instead of StatusValue

https://gerrit.wikimedia.org/r/586357

Change 586491 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Add WikiId in error messages when fetching page content

https://gerrit.wikimedia.org/r/586491

Change 587041 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Revert: Allow a page to be sent as message

https://gerrit.wikimedia.org/r/587041

We discovered some issues while testing this on Beta cluster, and other issues prevented us from knowing what exactly went wrong. To reduce risks, we are reverting our changes to avoid them going out with next train. We will continue testing after the new branch has been cut.

Change 587041 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Revert: Allow a page to be sent as message

https://gerrit.wikimedia.org/r/587041

Change 587210 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Revert "Revert: Allow a page to be sent as message"

https://gerrit.wikimedia.org/r/587210

Change 587251 had a related patch set uploaded (by Nikerabbit; owner: Nikerabbit):
[operations/mediawiki-config@master] Enable MassMessage logging on Beta Cluster

https://gerrit.wikimedia.org/r/587251

Change 587251 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MassMessage logging on Beta Cluster

https://gerrit.wikimedia.org/r/587251

Change 587210 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Revert "Revert: Allow a page to be sent as message"

https://gerrit.wikimedia.org/r/587210

Change 586357 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Use Status instead of StatusValue

https://gerrit.wikimedia.org/r/586357

Change 586491 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Add wikiId in log message when fetching page content from other wikis

https://gerrit.wikimedia.org/r/586491

Change 586353 abandoned by Nikerabbit:
Restore Beta Cluster logging

Reason:
This approach doesn't work.

https://gerrit.wikimedia.org/r/586353

Change 588368 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Fix call to wfGetDB, where wiki Id is the third parameter

https://gerrit.wikimedia.org/r/588368

Change 588368 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Fix call to wfGetDB, where wiki Id is the third parameter

https://gerrit.wikimedia.org/r/588368

Change 588441 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Use API instead of database when fetching content from another wiki

https://gerrit.wikimedia.org/r/588441

Change 588441 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Use API instead of database when fetching content from another wiki

https://gerrit.wikimedia.org/r/588441

abi_ added a comment.Apr 17 2020, 5:15 AM

We've added changes to allow users to select a page (including translatable page) and send that as a message. Next we need to add functionality to use fallback language.

We discovered some issues while testing this on Beta cluster, and other issues prevented us from knowing what exactly went wrong. To reduce risks, we are reverting our changes to avoid them going out with next train. We will continue testing after the new branch has been cut.

Whenever you'll test this in production, maybe you'll not want to use Tech News for it, given how problematic it usually is regardless? :)

Well, the issue was fixed and this is now deployed (sans fallback support). We would appreciate help testing this non-disruptively.

Change 591317 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Add fallback language support when sending translatable page as message

https://gerrit.wikimedia.org/r/591317

Well, the issue was fixed and this is now deployed (sans fallback support). We would appreciate help testing this non-disruptively.

The fallback support was /the/ request in this task. If you elaborate on what you need to be tested, maybe we could figure out a way to make it happen - cc @Johan.

What we have done so far is that you can now specify a translatable page to be send as the message. When message is delivered to a recipient page, it will be in the language of the page (aka 99% of cases language of the wiki) if that translation exists. This required quite a few code changes and has to deal with job queue and cross wiki stuff, which has a high probability of failing in unexpected ways.

The next step addresses the case where translation is not there. We will pick the first language that is available from this list:

  • target language
  • target language's fallbacks as defined in MediaWiki
  • source language of the translatable page

Due to deployment break, it will take a bit longer than usual until this is merged and in production. This change is less likely to fail in unexpected ways.

TL;DR: It's not a big deal, because of vacations etc, if testing is postponed until the full feature is available. But if early tests uncover issues, it can help us to resolve them faster,

Is there going to be an option if for whatever reason specific wikis need to be targeted in a fallback which is not the one defined in MediaWiki/of their choice? (IIRC this was controversial in the past, for political reasons and such.)

Not planned as part of this task.

Change 591317 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Add fallback language support when sending translatable page as message

https://gerrit.wikimedia.org/r/591317

I've to test this on the beta cluster.

Tested the fallback language functionality on the beta cluster and did not notice any issues.

The functionality implemented for this ticket has been documented under the Help page for the Mass Message extension.

Our "stress test" on MediaWiki.org failed:

09:06, 27 May 2020 Delivery of "Test message 2020-05-27@1" to User talk:APatro (WMF)/fr failed with an error code of The page "Help:Help:Extension:Translate/en", selected to be sent as message, was not found.
09:06, 27 May 2020 Delivery of "Test message 2020-05-27@1" to User talk:Nikerabbit failed with an error code of The page "Help:Help:Extension:Translate/en", selected to be sent as message, was not found.
09:06, 27 May 2020 Delivery of "Test message 2020-05-27@1" to User talk:APatro (WMF) failed with an error code of The page "Help:Help:Extension:Translate/en", selected to be sent as message, was not found.
09:06, 27 May 2020 Nikerabbit talk contribs block sent a message to User:Nikerabbit/T165128 (Test message 2020-05-27@1)

The namespace is present twice.

Change 599057 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Fix incorrect namespace usage when building the title

https://gerrit.wikimedia.org/r/599057

Change 599057 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Fix incorrect namespace usage when building the title

https://gerrit.wikimedia.org/r/599057

abi_ added a comment.Thu, May 28, 11:50 AM

Once the changes for this task are deployed, user's of the MassMessage extension will be able to, send a translated page from the wiki as a message. Depending on the language of the target page where the message has to be delivered, and whether a translation of the source page is available in that target page language, the source page will be either delivered in the target page language, a fallback language, or the source language.

Single line summary: Added support for sending multilingual pages with language fallbacks to the Mass Message extension

Nikerabbit updated the task description. (Show Details)Thu, May 28, 11:58 AM