We've reviewed how the Tech news gets sent out on a weekly basis. Here's a a summary of steps,

1. Prepare the tech news in English, and mark the page for translation.
2. Send a message on translators-l list requesting translators to translate it into different languages.
3. Translators submit translations via the Translate extension.
4. Generate this message from all the translations using the Lua script here.
5. Send the generated message using the MassMessage extension.

The message contains a switch case provided by the Parser function extension. So currently MassMessage does not care about the source language of the wiki, or the language of the message. It just takes the messages and pushes it to all the pages on given wiki. Thanks to @Johan for his inputs.

There are two possible ways to handle this,

1. Update the Lua script or add another parser function to handle fallback languages.
2. Allow MassMessage extension to take a translatable page as input for the message to be sent.

We're currently planning to implement option 2. This has the following benefits,

1. Simpler process for sending out the tech news, where step 4 above will no longer be needed.
2. Address or work towards addressing other issues such as,
   • T56839: More support for translations / messages in many languages
   • T70512: Please include an easy way to mark the language for the "Body of the message" to add automatic language tagging during delivery
   • T164503: Add feature to allow multilingual headings for translated MassMessage newsletters
   • T200414: Consider language variants when delivering content

Fallback languages can be determined by using the LanguageFallback class,

$service = MediaWikiServices::getInstance();
// => MediaWiki\MediaWikiServices {#15}
$service->getLanguageFallback();
// => MediaWiki\Languages\LanguageFallback {#139}
$lang = $service->getLanguageFallback();
// => MediaWiki\Languages\LanguageFallback {#139}
$lang->getAll( 'pt-br' )

Change 576850 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] [WIP] Add a message page field on the Special:MassMessage page

https://gerrit.wikimedia.org/r/576850

Change 578467 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Update jobs to process page-message parameter

https://gerrit.wikimedia.org/r/578467

Testing is blocked on actually getting the patches merged: T249194: Deprecated: Use of parserTest v1 in Scribunto and SyntaxHighlight
I gave myself necessary rights on the beta cluster and started https://meta.wikimedia.beta.wmflabs.org/wiki/T165128_testing_list

Change 576850 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Add a page selector on Special:MassMessage to be sent as message

https://gerrit.wikimedia.org/r/576850

Change 578467 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Update jobs to process the messages that contain a page

https://gerrit.wikimedia.org/r/578467

Change 586353 had a related patch set uploaded (by Nikerabbit; owner: Nikerabbit):
[operations/mediawiki-config@master] Restore Beta Cluster logging

https://gerrit.wikimedia.org/r/586353

Change 586357 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Use Status instead of StatusValue

https://gerrit.wikimedia.org/r/586357

Change 586491 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Add WikiId in error messages when fetching page content

https://gerrit.wikimedia.org/r/586491

Change 587041 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Revert: Allow a page to be sent as message

https://gerrit.wikimedia.org/r/587041

We discovered some issues while testing this on Beta cluster, and other issues prevented us from knowing what exactly went wrong. To reduce risks, we are reverting our changes to avoid them going out with next train. We will continue testing after the new branch has been cut.

Change 587041 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Revert: Allow a page to be sent as message

https://gerrit.wikimedia.org/r/587041

Change 587210 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Revert "Revert: Allow a page to be sent as message"

https://gerrit.wikimedia.org/r/587210

Change 587251 had a related patch set uploaded (by Nikerabbit; owner: Nikerabbit):
[operations/mediawiki-config@master] Enable MassMessage logging on Beta Cluster

https://gerrit.wikimedia.org/r/587251

Change 587251 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MassMessage logging on Beta Cluster

https://gerrit.wikimedia.org/r/587251

Change 587210 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Revert "Revert: Allow a page to be sent as message"

https://gerrit.wikimedia.org/r/587210

Change 586357 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Use Status instead of StatusValue

https://gerrit.wikimedia.org/r/586357

Change 586491 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Add wikiId in log message when fetching page content from other wikis

https://gerrit.wikimedia.org/r/586491

Change 586353 abandoned by Nikerabbit:
Restore Beta Cluster logging

Reason:
This approach doesn't work.

https://gerrit.wikimedia.org/r/586353

Change 588368 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Fix call to wfGetDB, where wiki Id is the third parameter

https://gerrit.wikimedia.org/r/588368

Change 588368 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Fix call to wfGetDB, where wiki Id is the third parameter

https://gerrit.wikimedia.org/r/588368

Change 588441 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Use API instead of database when fetching content from another wiki

https://gerrit.wikimedia.org/r/588441

Change 588441 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Use API instead of database when fetching content from another wiki

https://gerrit.wikimedia.org/r/588441

We've added changes to allow users to select a page (including translatable page) and send that as a message. Next we need to add functionality to use fallback language.

In T165128#6035204, @Nikerabbit wrote:

We discovered some issues while testing this on Beta cluster, and other issues prevented us from knowing what exactly went wrong. To reduce risks, we are reverting our changes to avoid them going out with next train. We will continue testing after the new branch has been cut.

Whenever you'll test this in production, maybe you'll not want to use Tech News for it, given how problematic it usually is regardless? :)

Well, the issue was fixed and this is now deployed (sans fallback support). We would appreciate help testing this non-disruptively.

Change 591317 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Add fallback language support when sending translatable page as message

https://gerrit.wikimedia.org/r/591317

In T165128#6074228, @Nikerabbit wrote:

Well, the issue was fixed and this is now deployed (sans fallback support). We would appreciate help testing this non-disruptively.

The fallback support was /the/ request in this task. If you elaborate on what you need to be tested, maybe we could figure out a way to make it happen - cc @Johan.

What we have done so far is that you can now specify a translatable page to be send as the message. When message is delivered to a recipient page, it will be in the language of the page (aka 99% of cases language of the wiki) if that translation exists. This required quite a few code changes and has to deal with job queue and cross wiki stuff, which has a high probability of failing in unexpected ways.

The next step addresses the case where translation is not there. We will pick the first language that is available from this list:

• target language
• target language's fallbacks as defined in MediaWiki
• source language of the translatable page

Due to deployment break, it will take a bit longer than usual until this is merged and in production. This change is less likely to fail in unexpected ways.

TL;DR: It's not a big deal, because of vacations etc, if testing is postponed until the full feature is available. But if early tests uncover issues, it can help us to resolve them faster,

Is there going to be an option if for whatever reason specific wikis need to be targeted in a fallback which is not the one defined in MediaWiki/of their choice? (IIRC this was controversial in the past, for political reasons and such.)

Not planned as part of this task.

Change 591317 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Add fallback language support when sending translatable page as message

https://gerrit.wikimedia.org/r/591317

I've to test this on the beta cluster.

Currently blocked due to T251510: Cannot find module '../lib/factories/wikimedia-eventgate'

Tested the fallback language functionality on the beta cluster and did not notice any issues.

The functionality implemented for this ticket has been documented under the Help page for the Mass Message extension.

Our "stress test" on MediaWiki.org failed:

09:06, 27 May 2020 Delivery of "Test message 2020-05-27@1" to User talk:APatro (WMF)/fr failed with an error code of The page "Help:Help:Extension:Translate/en", selected to be sent as message, was not found.
09:06, 27 May 2020 Delivery of "Test message 2020-05-27@1" to User talk:Nikerabbit failed with an error code of The page "Help:Help:Extension:Translate/en", selected to be sent as message, was not found.
09:06, 27 May 2020 Delivery of "Test message 2020-05-27@1" to User talk:APatro (WMF) failed with an error code of The page "Help:Help:Extension:Translate/en", selected to be sent as message, was not found.
09:06, 27 May 2020 Nikerabbit talk contribs block sent a message to User:Nikerabbit/T165128 (Test message 2020-05-27@1)

The namespace is present twice.

Change 599057 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/MassMessage@master] Fix incorrect namespace usage when building the title

https://gerrit.wikimedia.org/r/599057

Change 599057 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Fix incorrect namespace usage when building the title

https://gerrit.wikimedia.org/r/599057

Once the changes for this task are deployed, user's of the MassMessage extension will be able to, send a translated page from the wiki as a message. Depending on the language of the target page where the message has to be delivered, and whether a translation of the source page is available in that target page language, the source page will be either delivered in the target page language, a fallback language, or the source language.

Single line summary: Added support for sending multilingual pages with language fallbacks to the Mass Message extension

Tested on mediawiki.org and the previously failing test now passes.

TODO: Test remote-wiki delivery on metawiki and use of fallback language when target language is not present.

I spoke with @Johan to understand whether this can replace the current process of sending TechNews. One requirement which we are still missing, is ability to target a sections of a page (so that we do not deliver language list, categories, headers etc.).

I will create a new task about sending TechNews directly using Special:MassMessage, and mark this as a subtask of it, and add new requirements as separate subtasks.

Curiously, that was exactly the use case I wanted this done in the first place. Thanks team!

I don't know how to do this:
Post-deployment testing: try sending out a translatable page as a mass message, on MetaWiki, to different projects, on pages in different languages, with also testing fallbacks

I've gotten necessary access and will be taking care of testing this.

Thanks @Amire80 for creating the Message delivery list for testing this task. Tested on MetaWiki. Messages were sent from MetaWiki to MediaWiki.

1. Fallback languages were used properly. (pt was used as a fallback for pt-br)
2. If no translation is available, English is used as the final fallback.

The new functionality has been documented here.

Question: is there some way to use the fallback language's parent language? I.e. we frequently get translations in pt-br, and not in pt. When we deliver these massmessages, the Ptwikis all get the English version. We want to fix this situation.

pt falls back to pt-br and then en.

pt-br falls back to pt and then en.

So this should already work when the parent ticket is resolved and TechNews delivery process is updated to use Special:MassMessage directly.