Page MenuHomePhabricator

Emails send by subscribers don't arrive on the mailing list Moderators-nl
Closed, ResolvedPublic

Description

Without error messages none of the emails send by me - one of the subscribers - arrive on the mailing list Moderators-nl: https://lists.wikimedia.org/mailman/listinfo/moderators-nl. Only administrators on the Dutch Wikipedia are subscribed to this list. I'm also subscribed to other Wikimedia mailing lists such as checkuser-l and stewards-l and while it sometimes takes some time usually the emails arrive on the lists. But for some reason this is not the case for Moderators-nl. I asked list administrator Natuur12 to forward the email I send to him, but that didn't work either.

Some details: I use a hotmail.com email address and I tried to send two different emails - about different topics - on 26 November and on 2 December. Up til now both haven't arrived. Others also believe the list is 'broken' somehow. (For the record: the list did work before; the last email was received on 17 October.)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 2 2017, 10:58 PM
Trijnstel updated the task description. (Show Details)Dec 2 2017, 11:22 PM
Trijnstel updated the task description. (Show Details)Dec 2 2017, 11:40 PM

I haven't received them either. Did the emails show up in https://lists.wikimedia.org/mailman/private/moderators-nl/ ? (or did we disable the archive?)

Archives are disabeled and empty.

Trijnstel triaged this task as High priority.Dec 11 2017, 11:02 PM
Trijnstel added a subscriber: Platonides.

Did someone find the time to fix this? Platonides maybe? Up til now nothing happened... and I'd say it's pretty urgent.

herron added a subscriber: herron.Dec 12 2017, 8:20 PM

Hello, I've just adjusted the dmarc_moderation_policy for this list (under privacy options > sender filters) in an attempt to rule out DMARC as the cause of delivery issues from a hotmail address. Could you please check to see if this helps?

If the problem persists I would suggest temporarily enabling list archival for further troubleshooting and testing to see if the problem can be reproduced from a different email provider.

I just send two email to this list. One using my hotmail account and one using my gmail account. Both didn't arrive and both are subscribed to the list. I didn't receive any of the mails.

After this I enabeled the archives and send two more testmessages but the archives are still empty.

I did check my spambox on both occasions.

I just send two email to this list. One using my hotmail account and one using my gmail account. Both didn't arrive and both are subscribed to the list. I didn't receive any of the mails.
After this I enabeled the archives and send two more testmessages but the archives are still empty.
I did check my spambox on both occasions.

I can confirm that none of these emails arrived in my inbox.

Hello, I've just adjusted the dmarc_moderation_policy for this list (under privacy options > sender filters) in an attempt to rule out DMARC as the cause of delivery issues from a hotmail address. Could you please check to see if this helps?
If the problem persists I would suggest temporarily enabling list archival for further troubleshooting and testing to see if the problem can be reproduced from a different email provider.

Perhaps you could look at it again?

Multichill added a subscriber: RobH.

This seems to be going on too long without progress. @RobH can you help us out here? Should be easy to find on the commandline where in the flow <incoming mailserver> -> <mailman> -> <outgoing mailserver> -> subscribers the emails get lost. Help appreciated.

I've sent a test message through just now and see it in the list archive. Did you receive this message in your email?

To trace logs please try sending mail to the list again and follow up with the time sent and email address(es) used.

Your email arrived at 15:25 (Dutch local time), I send another email using a hotmail adress at 15:31 (I'm sorry but I'm not going to list my emailadress, Trijnstel or Multi could one off you please check if this one arrived) another user send an email using a gmail adress which arrived at 15:53. I send another mail using a gmail account at 16:00 and this one arrived at 16:00

Magere_Hein added a comment.EditedDec 22 2017, 11:41 AM

I tried to send a message twice from a gmail account to this mailing list earlier (12:09 CET and 12:10 CET), but that failed miserably: I did not receive a confirmation email and it didn't show up in the archive. A reply in the test topic sent at 12:06 CET did succeed, but a reply to that test at 12:15 CET again failed. This is both annoying and confusing.

Ciell added a subscriber: Ciell.Dec 23 2017, 8:50 PM

I received the message send by herron and by you Magere Hein.
This is strange, can it please be solved?

herron added a subscriber: Dzahn.Jan 4 2018, 9:44 PM

Just now I sent 3 testing messages with different subjects and 3 replies (one reply to each subject). The problem is happening for me as well. The reply to subject "testing3" was not delivered by email or archived.

The missing message did reach the lists MTA and the log indicates it was delivered to the mailman command, but there is no trace of the message in the mailman post log. So it seems the issue may lie somewhere in the handoff between MTA and mailman.

@RobH @Dzahn have you encountered similar symptoms before?

Just now I sent 3 testing messages with different subjects and 3 replies (one reply to each subject). The problem is happening for me as well. The reply to subject "testing3" was not delivered by email or archived.
The missing message did reach the lists MTA and the log indicates it was delivered to the mailman command, but there is no trace of the message in the mailman post log. So it seems the issue may lie somewhere in the handoff between MTA and mailman.
@RobH @Dzahn have you encountered similar symptoms before?

For the record: I sent a message from my Hotmail address again, and again that one didn't arrive....

Natuur12 added a comment.EditedJan 4 2018, 10:37 PM

I received five emails from Herron, zero from Trijnstel.

Checked a few mails I received directly from Trijnstel. Microsoft
recently used 40.92.65.66 to contact the post office of Google. An older
mail used 65.55.34.18. I presume both IPs are in a cloud, so other mails
may show a different IP from the same rather large ranges.

Could the cause of refusing hotmail be an IP related anti-spam block
somewhere between MS and Wikimedia? Just an idea.

Ronald

Dzahn added a comment.EditedJan 5 2018, 2:40 AM

@herron I don't know either, I don't remember an issue like this.

But..i saw there is this file called "legacy_mailing_lists" and the exim config for those seems to be different somewhat:

puppet: modules/role/files/exim/legacy_mailing_lists

file { '/etc/exim4/legacy_mailing_lists':
...
 source  => 'puppet:///modules/role/exim/legacy_mailing_lists',
# Alias old mailing list addresses to @lists.wikimedia.org
legacy_mailing_lists:
        driver = redirect
        domains = +legacy_mailman_domains
        data = $local_part$local_part_suffix@lists.wikimedia.org
        local_parts = lsearch;CONFDIR/legacy_mailing_lists
        local_part_suffix = -bounces : -bounces+* : \ 
                                -confirm+* : -join : -leave : \ 
                                -owner : -request : -admin : \ 
                                -subscribe : -unsubscribe
        local_part_suffix_optional

and moderators-nl IS on that list. Could that be related?

domainlist legacy_mailman_domains = wikimedia.org : wikipedia.org

Did any of these test emails maybe go to @wikipedia.org vs @wikimedia.org?

Dzahn added a comment.Jan 5 2018, 2:52 AM

Have you been able to contact the list administrator? Or is that one of you?

Moderators-nl list run by taketawiki at hotmail.com, l.haanstra at yahoo.com

The list admins should take a look at the anti-spam regex settings at: https://lists.wikimedia.org/mailman/admin/moderators-nl/privacy/spam

I suspect it might be due to these:

bounce_matching_headers : https://lists.wikimedia.org/mailman/admin/moderators-nl/?VARHELP=privacy/spam/bounce_matching_headers

header_filter_rules: https://lists.wikimedia.org/mailman/admin/moderators-nl/?VARHELP=privacy/spam/header_filter_rules

There appear to be many simple words in those anti-spam rules including the word Microsoft.

Because the needs for anti-spam rules are unique to each list this is something that should be done by the list administrators only.

herron added a comment.Jan 5 2018, 4:33 PM

But..i saw there is this file called "legacy_mailing_lists" and the exim config for those seems to be different somewhat:
puppet: modules/role/files/exim/legacy_mailing_lists

file { '/etc/exim4/legacy_mailing_lists':
...
 source  => 'puppet:///modules/role/exim/legacy_mailing_lists',
# Alias old mailing list addresses to @lists.wikimedia.org
legacy_mailing_lists:
        driver = redirect
        domains = +legacy_mailman_domains
        data = $local_part$local_part_suffix@lists.wikimedia.org
        local_parts = lsearch;CONFDIR/legacy_mailing_lists
        local_part_suffix = -bounces : -bounces+* : \ 
                                -confirm+* : -join : -leave : \ 
                                -owner : -request : -admin : \ 
                                -subscribe : -unsubscribe
        local_part_suffix_optional

and moderators-nl IS on that list. Could that be related?

I don't think it is since this configuration is on the mx[12]001 cluster while the mx for lists.wikimedia.org is the lists.wikimedia.org server itself.

The list admins should take a look at the anti-spam regex settings at: https://lists.wikimedia.org/mailman/admin/moderators-nl/privacy/spam
I suspect it might be due to these:
bounce_matching_headers : https://lists.wikimedia.org/mailman/admin/moderators-nl/?VARHELP=privacy/spam/bounce_matching_headers
header_filter_rules: https://lists.wikimedia.org/mailman/admin/moderators-nl/?VARHELP=privacy/spam/header_filter_rules
There appear to be many simple words in those anti-spam rules including the word Microsoft.
Because the needs for anti-spam rules are unique to each list this is something that should be done by the list administrators only.

This seems promising. When looking for logs related to this I also see many "bad regexp in bounce_matching_header" in the mailman config log. For example:

Jan 04 21:25:18 2018 (668) bad regexp in bounce_matching_header line: Moderators-nl

I'm one of the list admins . Those spamfilters are ancient and likely haven't been updated in a long, long time. I deleted all three spamfilters (but made a copy of the words used) and I have send a new testmail using my hotmail account. Sjoerd confirmed via IRC that he received this email. Though I do wonder why this problem occured now and not ages ago.

Jan 04 21:25:18 2018 (668) bad regexp in bounce_matching_header line: Moderators-nl

Are these mails dropped or can we expect a bunch of mails coming in because one of the mail servers queued them?

Just replied to the email and I got it within seconds so seems to work again. Thanks for your help @herron !

When was the last mailman update or configuration change? Maybe the spam filters broke around that time or the action changed what to do with an email hitting a broken spam filter?

herron added a comment.Jan 5 2018, 6:48 PM

Are these mails dropped or can we expect a bunch of mails coming in because one of the mail servers queued them?

These were dropped afaict

When was the last mailman update or configuration change? Maybe the spam filters broke around that time or the action changed what to do with an email hitting a broken spam filter?

The last mailman config change was on August 1, and the first occurrence of the "bad regexp" error was on Oct 11 2017

Well, seems that this is resolved. (Though we need a new spamfilter.) Thank you so much for all the help Herron and Dzahn.

Dzahn closed this task as Resolved.Jan 5 2018, 8:47 PM
Dzahn claimed this task.