Page MenuHomePhabricator

Translation Notification Bot sending the same message multiple times to every translator
Open, HighPublic

Description

Hello. It seems that Translation Notification Bot is broken again. It is sending the same notification up to 4 times to the translators. I've picked a random example at: https://meta.wikimedia.org/w/index.php?title=User_talk:MrLeopold&action=history. I've globally locked the account to prevent the bot from editting until this can be sorted out. Thanks.

Details

Related Gerrit Patches:
mediawiki/extensions/TranslationNotifications : masterAdd a new job to send out emails using UserMailer directly
mediawiki/extensions/TranslationNotifications : masterMove code to create translation notification jobs to a separate job
mediawiki/extensions/TranslationNotifications : masterUse MassMessage to send notification to user's talkpage
mediawiki/extensions/TranslationNotifications : masterBreakout notification sending logic into a separate class
integration/config : masterAdd MassMessage as a dependency for TranslationNotifications
operations/mediawiki-config : masterFix incorrect channel name for TranslationNotifications extension
mediawiki/extensions/TranslationNotifications : masterAdd logging to the TranslationNotificationJob
operations/mediawiki-config : masterAdd channels for the Translate and TranslationsNotification extension

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Putting up for consideration for a sprint.

Any progress here? Thanks.

This task is not currently classified as UBN!, and per Language-Team's repository maintenance list we only look at UBN!s for this extension. This task will be considered when there is a focus on maintenance.

MarcoAurelio raised the priority of this task from High to Unbreak Now!.Feb 28 2017, 12:04 PM

In that case I'm changing this as UBN priority since this is something that worked in the past correctly and is now working with mistakes. Also because it's hindering the translation work as translators are not being aware of new translation requests, even for important events. Thanks for your understanding. Best regards.

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptFeb 28 2017, 12:04 PM
Nemo_bis lowered the priority of this task from Unbreak Now! to High.Feb 28 2017, 1:48 PM
Nemo_bis added a subscriber: Nemo_bis.

I would like the Language-Team policy to change, but this task doesn't qualify for "unbreak now": I'm not seeing any critical disruption right now. For instance, https://meta.wikimedia.org/wiki/Special:Contributions/Translation_Notification_Bot doesn't show any flood of duplicate messages and translation notifications can be sent normally, although a minority of translators might suffer an annoyance.

I tested it today after a year of block. The problem seems to be when several translators are to be notified, and translation admins, after the page erroring, reload. The bot then resends the same notifications again. I think that after you hit "send", the page that says that the notifications have been queued should appear and let the background jobs run w/o user awareness, so we can avoid the issue.

Yes, that's a common mistake that adds noise to our data points. However in some cases the bug is real and goes along the lines described by T144780#2611657.

I've seen the sign on the special page AFTER clicking on the "send button" (and of course, after previewed my message multiple times). Instead of leaving a banner that no one notices (banners are like the logo ot the left bar: you mentally skip them), would it be possible to temporary disable the feature?

@Trizek-WMF Disabling is possible, absent a better method, setting this it to false on InitialiseSettings.php. However that does not resolve the issues mentioned here. I wonder how MassMessage works for this kind of mass sending jobs and maybe we can copy from there? Regards.

Indeed I don't see how disabling the feature would help the goal, i.e. have the translators receive the information they need asked for with minimum collateral damage.

Maybe it's a silly idea, but how if translation notifications, instead of leaving TP/Email, triggered a notification?

I was suggesting disabling, because the message explicitly says "don't use the form to inform translators".

I see it as the same thing as put a not a a button expecting people not to use it versus disconnecting the wires to avoid the button to work, or, better, remove the button (and leave a note). But again, IANAD (I'm not a developer). ;)

Misleading messages should be removed but they're not a coding matter.

What is misleading in Please do not use this feature until T144780 is resolved. Thank you!, please? https://meta.wikimedia.org/wiki/MediaWiki:Translationnotifications-text-fetchi18n

That box should always appear in Special:NotifyTranslators. If it was not in your case, then we should investigate why.

Again, disabling is an option, but it's not going to resolve the underlying issues.

I'll file a task to convert the system into using Echo notifications, at least let it be investigated.

What is misleading in Please do not use this feature until T144780 is resolved. Thank you!, please? https://meta.wikimedia.org/wiki/MediaWiki:Translationnotifications-text-fetchi18n
That box should always appear in Special:NotifyTranslators. If it was not in your case, then we should investigate why.

As reported, I've seen it, but too late. People tend to skip warnings as they are to common on the wikis.

Again, disabling is an option, but it's not going to resolve the underlying issues.

Sure. But if it is done temporary, it would prevent possible problems.

Billinghurst added a subscriber: Billinghurst.EditedFeb 2 2018, 7:42 AM

This task is not currently classified as UBN!, and per Language-Team's repository maintenance list we only look at UBN!s for this extension. This task will be considered when there is a focus on maintenance.

Happy 1 year birthday to comment that we do not UBN. Is this any closer to being fixed?

Seems unusual to have a notification that isn't used as it somewhat sends duplicate messages. So there is a backlog of untranslated pages that need translating. Is the implicit decision that no prodding for translations is better than the occasional duplication of messages?

Does this multiple sending problem exist on other wikis with the same translation system, or is it just meta?

Hi @Nikerabbit ! I was wondering if you could take a look at this ticket and work on the notify translators tool? At the moment, it sends 4 messages to each translator, for every message you send. Since we are working on the second iteration of Wikimedia communities and contributors survey, and we need to have some messages translated for the launch. It would be great to be able to use this tool for that.

Thanks for all the work!

@mcruzWMF - Where did the bot sent 4 messages? Was this recently? If so, on which wiki? Thanks.

Hi @MarcoAurelio sorry for the late response. This was on Meta-Wiki, I believe. I haven't tried the tool recently, this was at the end of February, beginning of March.

Tomybrz added a subscriber: Tomybrz.

Please note that, specially on Meta-Wiki, using Special:NotifyTranslators usually results on DB timeouts for languages where a large number of users signed up to be notified via Special:TranslatorSignup. I guess we need to batch the requests/queries. Thanks.

Change 536738 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/TranslationNotifications@master] Adding logging to the TranslationNotificationJob

https://gerrit.wikimedia.org/r/536738

Change 537250 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/TranslationNotifications@master] Breakout notification sending logic into a separate class

https://gerrit.wikimedia.org/r/537250

Change 537363 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[operations/mediawiki-config@master] Add channels for the Translate and TranslationsNotification extension

https://gerrit.wikimedia.org/r/537363

Change 537363 merged by jenkins-bot:
[operations/mediawiki-config@master] Add channels for the Translate and TranslationsNotification extension

https://gerrit.wikimedia.org/r/537363

Mentioned in SAL (#wikimedia-operations) [2019-09-17T11:31:17Z] <urbanecm@deploy1001> Synchronized wmf-config/VariantSettings.php: SWAT: 290e207: Add channels for the Translate and TranslationsNotification extension (T221119, T144780, T143073) (duration: 00m 56s)

Change 537575 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/TranslationNotifications@master] Move code for creating notification jobs to a seperate job

https://gerrit.wikimedia.org/r/537575

Change 537628 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[operations/mediawiki-config@master] Fix incorrect channel name for TranslationNotifications extension

https://gerrit.wikimedia.org/r/537628

Change 536738 merged by jenkins-bot:
[mediawiki/extensions/TranslationNotifications@master] Add logging to the TranslationNotificationJob

https://gerrit.wikimedia.org/r/536738

Change 537628 merged by jenkins-bot:
[operations/mediawiki-config@master] Fix incorrect channel name for TranslationNotifications extension

https://gerrit.wikimedia.org/r/537628

Mentioned in SAL (#wikimedia-operations) [2019-09-25T16:06:40Z] <kartik@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit|537628|Fix incorrect channel name for TranslationNotifications extension (T144780)]] (duration: 01m 06s)

Change 539490 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/TranslationNotifications@master] [WIP] Use MassMessage to send notification to user's talkpage

https://gerrit.wikimedia.org/r/539490

Change 539547 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[integration/config@master] Add MassMessage as a dependency for TranslationNotifications

https://gerrit.wikimedia.org/r/539547

Change 539547 merged by jenkins-bot:
[integration/config@master] Add MassMessage as a dependency for TranslationNotifications

https://gerrit.wikimedia.org/r/539547

Moving this to in review as there are a series of patches submitted that we think should fix the issue. We will put it back into progress if we notice that the issue is continuing to happen.

Related patches,

  1. Breakout notification sending logic into a separate class
  2. Move code to create translation notification jobs to a seperate job
  3. Use MassMessage to send notification to user's talkpage

Change 537250 merged by jenkins-bot:
[mediawiki/extensions/TranslationNotifications@master] Breakout notification sending logic into a separate class

https://gerrit.wikimedia.org/r/537250

Change 537575 merged by jenkins-bot:
[mediawiki/extensions/TranslationNotifications@master] Move code to create translation notification jobs to a separate job

https://gerrit.wikimedia.org/r/537575

Change 539490 merged by jenkins-bot:
[mediawiki/extensions/TranslationNotifications@master] Use MassMessage to send notification to user's talkpage

https://gerrit.wikimedia.org/r/539490

These changes have now been deployed on all the wiki's.

Here's what we've done.

  1. Improved logging. Logs will now be written into logstash, and can be viewed via the Kibana dashboard.
  2. We've moved the following code to a separate job that does not hold up the web request,
    1. Checks whom notifications have to be sent.
    2. Create the notification object.
    3. Send the notification.
  3. We're now using the MassMessage extension to send out the notification to user's talk page.

We've sent an email on translator-l mailing list asking for feedback regarding the new changes.

Xiplus added a subscriber: Xiplus.Tue, Jan 14, 11:16 AM

Thanks @abi_ - I'll run some tests at Meta and report back if there are further issues.

@abi_ It looks it is not working. I sent a notification on meta.wikimedia (previosly I signed up as a translator for lb). After waiting for a bit I got no notification neither on my email nor on my talk page. Could you please check logstash and see what might have gone wrong? Or is it a jobQueue/caching issue and takes a while? Best regards.

@MarcoAurelio asked me to look into the logs, and I found TranslationNotificationSubmitJob does its work, but can't find anything
for TranslationNotificationJob. I grepped JobExecutor.log, and only TranslationNotificationSubmitJo found as well.

abi_ added a comment.Fri, Jan 17, 3:45 AM

Thanks @MarcoAurelio and @Urbanecm. I'll start investigating it.

abi_ added a comment.EditedFri, Jan 17, 5:30 AM

Following are the details for the request,

reqId: XiDOGQpAICMAALtmgXkAAACJ

I see two issues,

1. Non-scalar value found in the event

Channel: EventBus,
Message: Non-scalar value found in the event.
Data:

[
    {
        "$schema": "/mediawiki/job/1.0.0",
        "meta": {
            "uri": "https://placeholder.invalid/wiki/Special:Badtitle",
            "request_id": "XiDOGQpAICMAALtmgXkAAACJ",
            "id": "402fb40e-3678-4294-b200-b72eb9f94d28",
            "dt": "2020-01-16T20:57:00Z",
            "domain": "meta.wikimedia.org",
            "stream": "mediawiki.job.sendMail"
        },
        "database": "metawiki",
        "type": "sendMail",
        "params": {
            "to": {
                "name": "MarcoAurelio",
                "realName": "",
                "address": "<<removed>>"
            },
            "from": {
                "name": "MarcoAurelio",
                "realName": "",
                "address": "<<removed>>"
            },
            "body": "<<removed>>",
            "subj": "<<removed>>",
            "replyto": {
                "name": "MarcoAurelio",
                "realName": "",
                "address": "<<removed>>"
            },
            "namespace": 0,
            "title": "Main_Page",
            "requestId": "XiDOGQpAICMAALtmgXkAAACJ"
        }
    }
]

There was a similar issue here - T192945: Make EchoNotification job JSON-serializable

Need to investigate this further.

2. TranslationNotificationJob did not trigger

Originating wiki: metawiki

The TranslationNotificationSubmitJob seems to have run without any issues and found the following notifications to be sent,

  1. Wiki ID: dewiki, 1 notification / job
  2. Wiki ID: metawiki, 3 notifications / job

I subsequently don't see anything from TranslationNotificationJob in the logs. I don't see anything related to the requestId either. I'm not sure if this is because of the previous issue.

MarcoAurelio added a comment.EditedFri, Jan 17, 9:15 PM

Thanks @abi_ for the analysis. I assume there's logging in place for TranslationNotificationJob too? Do you think this is fixable? Best regards.

Also note that regarding the test translation notification that I triggered I subscribed to both email and talk page message, none of which did worked.

abi_ added a comment.Mon, Jan 20, 12:01 PM

We believe that the reason for failure is same as described here - T192945: Make EchoNotification job JSON-serializable . EmaillingJob accepts MailAddress class object as a parameters which is not JSON serializable and hence not supported by kafka-based queue.

We're planning to create another job and call UserMailer::send there directly.

We also believe that TranslationNotificationJob was not queued because of the initial failure with the EmaillingJob, and hence no logs.

Change 566079 had a related patch set uploaded (by Abijeet Patro; owner: Abijeet Patro):
[mediawiki/extensions/TranslationNotifications@master] Add a new job to send out emails using UserMailer directly

https://gerrit.wikimedia.org/r/566079