Page MenuHomePhabricator

Translation notifications via email digest is broken
Closed, ResolvedPublic2 Estimated Story PointsPRODUCTION ERROR

Description

The translationnotifications mailer is broken, which is preventing translators to receive emails and digests about pending translator tasks.

As far as I was told, the issue is with MailAddress but I'm seeing other errors with JobExecutor and sendEmail.

EventBus reports errors because of a Non-scalar value found in the event. According to @Ottomata in -analytics today this is because:

<ottomata> to field is being set an instance of MailAddress
<ottomata> which is failing because MailAddress is not a scalar
<ottomata> MailAddress's toString does work as expected though

Looking for related errors in Logstash I can find as well 4,000+ JobExecutor sendMail errors since January 2023. See this Logstash query e.g.:

"Failed executing job: sendMail Special:MyLanguage/Main_Page to={\"name\":\"***REDACTED***\",\"realName\":\"\",\"address\":\"\"} from={\"name\":\"\",\"realName\":\"\",\"address\":\"wiki@wikimedia.org\"} body=string(1324) subj=Digest email for translation requests from Meta replyto={\"name\":\"\",\"realName\":\"\",\"address\":\"***REDACTED***\"} namespace=-1 title=MyLanguage/Main_Page requestId=9be0ac0706badf0d293ec762",
Error
normalized_message
Non-scalar value found in the event
exception.trace
Impact
Notes
Original Task Description

I have opted-in to translation notifications via a weekly email digest for Spanish via Meta-Wiki Special:TranslatorSignup for quite some time. However, and despite translation notifications been sent according to the relevant on-wiki log, I have received no email or digests, which makes me suspect that either the whole "email" feature is broken, or maybe just the digest option is not working.

Just in case, my email address is valid and verified/confirmed as well.

Thank you.

Event Timeline

I don't see anything logged in TranslationNotifications.Jobs after 2023-03-22 13:49:10, which is a Wednesday (train deployment date to Meta). According to SAL, train was deployed to group1 at 18:16 that day. No changes in TranslateNotifications in wmf.1 though.

The last log entry at the time I'm writing this is from 13:48, 22 March 2023 (UTC). However, it's been probably a year or thereabouts that I opted to receive a weekly email digest and have been receiving none.

MarcoAurelio renamed this task from Translation notifications via email may be broken to Translation notifications via email digest may be broken.Apr 3 2023, 3:35 PM
MarcoAurelio updated the task description. (Show Details)

I changed my subscription from weekly digests to regular email, and I am receiving these; so I suspect a bug for the weekly/monthy digest options.

Looking at the logs kindly provided in T333851 the DigestEmailer.php script and systemd timer seems to be running and not erroring. However, despite reporting that it has sent digests to me since at least September 2021 I did never get any of them. My email client is not sending wikimedia.org to the spam folder either, nor I have any filter/rule that deletes these right away. I was adviced this may need someone in serviceops & Mail to go investigate deeper.

I think it's indeed broken. See e.g. this Logstash reqId entry for a digest I was suposed to receive.

EventBus complains of a Non-scalar value found in the event when running the systemd timer.

MarcoAurelio renamed this task from Translation notifications via email digest may be broken to Translation notifications via email digest is broken.Apr 19 2023, 4:35 PM
MarcoAurelio triaged this task as High priority.
MarcoAurelio updated the task description. (Show Details)
MarcoAurelio updated the task description. (Show Details)
MarcoAurelio added a subscriber: Ottomata.
MarcoAurelio changed the subtype of this task from "Bug Report" to "Production Error".Apr 19 2023, 4:49 PM
MarcoAurelio moved this task from Backlog to Bugs on the TranslationNotifications board.

image.png (494×1 px, 38 KB)

I see this issue started happening on Feb 6th: Logstash link. Nothing in the DigestEmailer.php has changed since last year. DigestEmailer.php is a fairly independent script, and the error does not feel related to a change inside the TranslationNotifications extension.

Nothing seems to have changed in MailAddress.php class either in a long time.

On Feb 6th, MediaWiki_1.40/wmf.21 was active. Here are the changelogs: https://www.mediawiki.org/wiki/MediaWiki_1.40/wmf.21 from that week. I can't find a suspect that would cause these issues to occur.

Change 916518 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/TranslationNotifications@master] DigestEmailer: Replace EmailingJob with TranslationNotificationsEmailJob

https://gerrit.wikimedia.org/r/916518

Change 916518 merged by jenkins-bot:

[mediawiki/extensions/TranslationNotifications@master] DigestEmailer: Replace EmailingJob with TranslationNotificationsEmailJob

https://gerrit.wikimedia.org/r/916518

DigestEmailer.php is a fairly independent script, and the error does not feel related to a change inside the TranslationNotifications extension.

I think I agree this is possibly failing "in transit". The script seems to submit the jobs correctly, but these emails never get sent. JobQueue rejecting and/or ditching emails?

As a "weekly digest" subscriber for a couple of years, I can attest I have not received one single digest in all this time, but I don't know the architecture and code well enough to know what may be causing this.

Looking for the last Digest I should've received on Meta, I can see:

  • EventBus: Non-scalar value found in the event.
  • JobExecutor: Failed executing job: sendMail [...].
  • Exec error in cpjobqueue for the changeprop-jobqueue for RetryExecutor.

So, it'd appear the JobQueue has issues with the data this extensions sends there for processing?

I tried to test the patches above through a test notification but logstash keeps complaining about non-scalar values (see logstash · by reqId).

The digest, if any, would be sent the next time the systemd script runs on Monday, unless you'd like to run the script manually and see what's the output.

prop_name: languageSet
prop_val_type: MediaWiki\Extension\TranslationNotifications\Utilities\LanguageSet

Change 920663 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/TranslationNotifications@master] SpecialNotifyTranslators: Json serialize the LanguageSet parameter

https://gerrit.wikimedia.org/r/920663

Change 920663 merged by jenkins-bot:

[mediawiki/extensions/TranslationNotifications@master] SpecialNotifyTranslators: Json serialize the LanguageSet parameter

https://gerrit.wikimedia.org/r/920663

I tried to test the patches above through a test notification but logstash keeps complaining about non-scalar values (see logstash · by reqId).

The digest, if any, would be sent the next time the systemd script runs on Monday, unless you'd like to run the script manually and see what's the output.

I sent another notification today that seems to have gone through well after the latest fixes. See: log link. logstash req Id. I also received the notification on my talk page.

I tried to test the patches above through a test notification but logstash keeps complaining about non-scalar values (see logstash · by reqId).

The digest, if any, would be sent the next time the systemd script runs on Monday, unless you'd like to run the script manually and see what's the output.

I ran the DigestEmailer.php manually today. According to the script output, you should have received an email. Please let me know if you did.

Hello @abi_ - I've received a digest last week, as well as this one :)

The strings The priority of this page is $1 and The deadline for translating this page is $1 appear untranslated for Spanish in the email body, although these are translated.

Can you file that as a separate bug? It's a pre-existing issue.