@Xover I created https://phabricator.wikimedia.org/P59624 that is restricted to WMF-NDA members and the subscribers of the paste (currently you), which should be secure to put the full headers

In T361860#9691323, @DannyS712 wrote:

@Xover I created https://phabricator.wikimedia.org/P59624 that is restricted to WMF-NDA members and the subscribers of the paste (currently you), which should be secure to put the full headers

Thanks. But Phab still won't let me edit that paste. Only members of WMF-NDA and the author have the "Can Edit" capability; and since normal users can't create private pastes that makes them pretty useless for reporting stuff like this. @Aklapper is this by design, or just permissions accidentally set too tightly?

PS. I got one more copy yesterday, but no new ones for the last 9 hours.

For ceeations, I think we just need to add you into phab trusted-contribs, I can do it when I'm back at my laptop if someone doesn't do it before me

In T361860#9691600, @Xover wrote:

@Aklapper is this by design, or just permissions accidentally set too tightly?

@Xover: Hi, please bring up general Phab questions on https://www.mediawiki.org/wiki/Talk:Phabricator/Help - thanks!

And now another two ticked in.

In T361860#9691600, @Xover wrote:

In T361860#9691323, @DannyS712 wrote:

@Xover I created https://phabricator.wikimedia.org/P59624 that is restricted to WMF-NDA members and the subscribers of the paste (currently you), which should be secure to put the full headers

Thanks. But Phab still won't let me edit that paste. Only members of WMF-NDA and the author have the "Can Edit" capability...

Oh, duh - try now

@Xover if you could paste the headers of two of the messages that would help, the whole raw text would be the best however. I am curious how identical the emails are, as that would help indicate where they are being duplicated.

In T361860#9693188, @jhathaway wrote:

@Xover if you could paste the headers of two of the messages that would help, the whole raw text would be the best however. I am curious how identical the emails are, as that would help indicate where they are being duplicated.

I added the raw text of the first of these resent messages to the paste (thanks Danny!), and the latest (as of 15 min. ago). The message body is bit for bit identical. In the headers I see only the expected transport differences. Interestingly, the Date and Message-ID differ. Message-ID is often generated by the MTA, but the Date field is usually generated by the MUA so that points at something at least before Exim having the hickup (but maybe Exim is just your outbound edge?).

Note that I redacted the final destination address from the headers. The pobox.com email I have set in MediaWiki forwards to a non-public gmail.com account and I have removed the username from this address.

I am the original recipient of the message by Xover discussed above. I confirm that I also still keep getting the message again and again. So far it has been resent to me 18 times. It is quite annoying not only because of the repeated messages themselves, but also because it floods my alerts in the Wikimedia projects.

In T361860#9694838, @Jan.Kamenicek wrote:

… it floods my alerts in the Wikimedia projects.

@Jan.Kamenicek Are you actually getting notifications ala. "User Xover sendt you an email" up in the Echo notification area, of the same kinds you get if someone pings you on wiki?

SRE / @jhathaway That makes it highly unlikely to me that this is a typical email hiccup down in the mail transport somewhere (like exim or postfix not cleaning out its spool file or whatever). I don't know what the plumbing is like for email, but for anything that actually generates an Echo notification it seems overwhelmingly likely that this is up in the application part of the stack. A database entry that is never getting tagged as completed; or a job queue job that is stuck and getting reprocessed; an internal message queue (like AMQP or whatever) ditto. Etc.

@Xover Yes, that is exactly the notification I get repeatedly alongside with each repeated email.

But now I haven't gotten any more copies since April 5, so whatever it was seems to have cleared for now. It's probably still a good idea to dig through the logs to get an idea what caused this since it could very well happen again and at larger scale.

Confirm, neither have I. Hope it will not recur.

And now I just got a resend of a different email to a different user, originally sent on April 11. That’s something like two out of three emails I’ve sent using «Email this user» the last month+ that are getting resent.

So either there is something very weird going on specifically with my account, or this is affecting a significant proportion of emails sendt using this function.

Given that this has reoccurred and from the emails you provided looks to be duplication on the application layer I think we need to rope in some people on the Mediawiki side. @larissagaulia do you know who might be a first point of contact?