Page MenuHomePhabricator

Old "Email this user" email is repeatedly resent
Open, HighPublicBUG REPORT

Description

On March 24 I used "Email this user" on enWS to email another user, asking for a copy. Today I've received 9 (and counting) copies of that email, and the original recipient reports getting the same. It started around 08:15 UTC today, and last came in 10 minutes ago.

That sounds to me like a hickup either down in the email transport (MTA), or in the jobqueue / cron / whatever that connects MediaWiki with the email transport. I haven't seen any other users reporting the same problem on-wiki, so I don't know if it's just this message or a general problem. But it could point at an awful lot of email getting resent to an awful lot of users.

I unfortunately am not able to create a private paste (it seems you need special permissions to do that) with the full email headers. But the only obviously relevant datum in there before the email hits pobox.com's MXes (who then forward it on to my GMail account) is that the sending server is "wiki-mail-eqiad.wikimedia.org" coming on IP "208.80.154.91" and is reporting its sender to be "wiki-enwikisource-[redacted]@wikimedia.org". SPF etc. all report ok, and the Received headers do not indicate this is going through any unexpected systems.

The Date header reports today's date and the time the message was actually sent (i.e. 17:02 UTC for the last message).

Event Timeline

RLazarus triaged this task as High priority.Apr 4 2024, 6:18 PM
RLazarus subscribed.

Clinic duty SRE here -- I/F, can you start investigating this at the MTA end? Triaging this to High in case it's widespread, but feel free to decrease if it turns out it's not.

@Xover I created https://phabricator.wikimedia.org/P59624 that is restricted to WMF-NDA members and the subscribers of the paste (currently you), which should be secure to put the full headers

@Xover I created https://phabricator.wikimedia.org/P59624 that is restricted to WMF-NDA members and the subscribers of the paste (currently you), which should be secure to put the full headers

Thanks. But Phab still won't let me edit that paste. Only members of WMF-NDA and the author have the "Can Edit" capability; and since normal users can't create private pastes that makes them pretty useless for reporting stuff like this. @Aklapper is this by design, or just permissions accidentally set too tightly?

PS. I got one more copy yesterday, but no new ones for the last 9 hours.

For ceeations, I think we just need to add you into phab trusted-contribs, I can do it when I'm back at my laptop if someone doesn't do it before me

@Aklapper is this by design, or just permissions accidentally set too tightly?

@Xover: Hi, please bring up general Phab questions on https://www.mediawiki.org/wiki/Talk:Phabricator/Help - thanks!

And now another two ticked in.

@Xover I created https://phabricator.wikimedia.org/P59624 that is restricted to WMF-NDA members and the subscribers of the paste (currently you), which should be secure to put the full headers

Thanks. But Phab still won't let me edit that paste. Only members of WMF-NDA and the author have the "Can Edit" capability...

Oh, duh - try now

@Xover if you could paste the headers of two of the messages that would help, the whole raw text would be the best however. I am curious how identical the emails are, as that would help indicate where they are being duplicated.

@Xover if you could paste the headers of two of the messages that would help, the whole raw text would be the best however. I am curious how identical the emails are, as that would help indicate where they are being duplicated.

I added the raw text of the first of these resent messages to the paste (thanks Danny!), and the latest (as of 15 min. ago). The message body is bit for bit identical. In the headers I see only the expected transport differences. Interestingly, the Date and Message-ID differ. Message-ID is often generated by the MTA, but the Date field is usually generated by the MUA so that points at something at least before Exim having the hickup (but maybe Exim is just your outbound edge?).

Note that I redacted the final destination address from the headers. The pobox.com email I have set in MediaWiki forwards to a non-public gmail.com account and I have removed the username from this address.

I am the original recipient of the message by Xover discussed above. I confirm that I also still keep getting the message again and again. So far it has been resent to me 18 times. It is quite annoying not only because of the repeated messages themselves, but also because it floods my alerts in the Wikimedia projects.

… it floods my alerts in the Wikimedia projects.

@Jan.Kamenicek Are you actually getting notifications ala. "User Xover sendt you an email" up in the Echo notification area, of the same kinds you get if someone pings you on wiki?

SRE / @jhathaway That makes it highly unlikely to me that this is a typical email hiccup down in the mail transport somewhere (like exim or postfix not cleaning out its spool file or whatever). I don't know what the plumbing is like for email, but for anything that actually generates an Echo notification it seems overwhelmingly likely that this is up in the application part of the stack. A database entry that is never getting tagged as completed; or a job queue job that is stuck and getting reprocessed; an internal message queue (like AMQP or whatever) ditto. Etc.

@Xover Yes, that is exactly the notification I get repeatedly alongside with each repeated email.

But now I haven't gotten any more copies since April 5, so whatever it was seems to have cleared for now. It's probably still a good idea to dig through the logs to get an idea what caused this since it could very well happen again and at larger scale.

Confirm, neither have I. Hope it will not recur.

And now I just got a resend of a different email to a different user, originally sent on April 11. That’s something like two out of three emails I’ve sent using «Email this user» the last month+ that are getting resent.

So either there is something very weird going on specifically with my account, or this is affecting a significant proportion of emails sendt using this function.

Given that this has reoccurred and from the emails you provided looks to be duplication on the application layer I think we need to rope in some people on the Mediawiki side. @larissagaulia do you know who might be a first point of contact?