Just to provide more complete information - approximately 1200 emails were received from the *@qq.com email addresses between 1330 UTC and 1634 UTC (when most of the list admins were removed as a stopgap measure, to stop the flooding of our personal email inboxes). They started at a rate of 1-2/minute and by the time I stopped receiving, they were coming at a rate of 15/minute. As best I can tell, this is the only *-L-owner mailing list affected - I am also a listadmin on a bunch of other lists and none of them were affected.

I have retained these mails temporarily, but plan to delete them on July 15 (they take up a lot of space, even if they're in the bin/spam folders), so if you need more examples or more information, please let me know soon.

Further note - being reported on other -owner lists via the List Admins mailing list.

As a listadmin of meta-oversight, I confirm that I'm also receiving frequent spam to the list and the -owner address; although from different domains. Happy to share the domains if needed, although I clean my spam folder very regularly.

Recently in T161082 a default_bounce_matching_headers setting was deployed to hold messages with a 6 or greater spam score. I think this default is overridden when a list specific bounce_matching_headers setting is present (as is the case with checkuser-l and wikials-l). I went ahead and added these lines to the wikials-l bounce_matching_headers regexp:

from: .*@qq.com
X-Spam-Score:[^+]*[+]{6,}

For other lists IMHO it would be worth either a) clearing the lists bounce_matching_headers list override to inherit the default, or b) add to bounce_matching_headers something like this:

X-Spam-Score:[^+]*[+]{6,}

From https://wikitech.wikimedia.org/wiki/Mailman

Spam scores

The mailman UI supports this via the configuration variable header_filter_rules aka. 'Spam Filter Regexp' (description: Filter rules to match against the headers of a message.). See also https://www.gnu.org/software/mailman/mailman-admin/sender-filters.html

This can be found in the administrative interface in Privacy options...-> [Spam filters] -> Spam Filter Regexp (or visit directly the URL, replacing YOURLIST with your list name: https://lists.wikimedia.org/mailman/admin/YOURLIST/?VARHELP=privacy/spam/header_filter_rules ).

So someone from SRE has to check Spam Filter Regexp for each list ? Write a small script ? Directly from the database ?

In T170601#3437121, @Framawiki wrote:

From https://wikitech.wikimedia.org/wiki/Mailman

Spam scores

The mailman UI supports this via the configuration variable header_filter_rules aka. 'Spam Filter Regexp' (description: Filter rules to match against the headers of a message.). See also https://www.gnu.org/software/mailman/mailman-admin/sender-filters.html

This can be found in the administrative interface in Privacy options...-> [Spam filters] -> Spam Filter Regexp (or visit directly the URL, replacing YOURLIST with your list name: https://lists.wikimedia.org/mailman/admin/YOURLIST/?VARHELP=privacy/spam/header_filter_rules ).

So someone from SRE has to check Spam Filter Regexp for each list ? Write a small script ?

Someone with a better core technical understanding of mailman may correct me but my understanding is that while this would help for the mailing lists (and so may be useful) one of the other big issues here (the spam being sent to list OWNERS) would not be resolved. I don't think Mailman has any spam setting options for -owner addresses.

Perhaps we should lower the spam score that results in a deny at the lists MTA from 12 to 6. It would require a change in only one place and act as a default for lists addresses. List admins could still optionally set up additional filters through mailman admin.

In T170601#3437121, @Framawiki wrote:

So someone from SRE has to check Spam Filter Regexp for each list ? Write a small script ? Directly from the database ?

I don't see why "someone from SRE" has to do that hence removing tag.

On Jul 14, 2017, at 2:34 AM, Manuel Schneider <manuel.schneider@wikimedia.ch> wrote:

sorry, but I received another 600 such mails over night...

I think the point is that you added a filter to the list but these mails
should be bounced by the MTA right away. In this case the mails aren't
directed at the list but at the -owner alias, so I think Mailman filters
are not effective here.

/Manuel

Change 365267 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Change lists to reject spam score of 6 or greater via exim acl

https://gerrit.wikimedia.org/r/365267

Change 365267 merged by Herron:
[operations/puppet@production] Change lists to reject spam score of 6 or greater via exim acl

https://gerrit.wikimedia.org/r/365267

Mentioned in SAL (#wikimedia-operations) [2017-07-14T16:36:46Z] <herron> lowered mailman/lists spam_score exim acl to 6 - T170601

365267 has been merged and I'm seeing loads of 'rejected after DATA' events for qq.com addresses in the exim logs now.

Change 365279 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Lists: Add exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/365279

Change 365279 merged by Herron:
[operations/puppet@production] Lists: Add exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/365279

Mentioned in SAL (#wikimedia-operations) [2017-07-14T18:54:16Z] <herron> added exim from/subject filter for spam observed from qq.com - T170601

Change 365424 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Lists: Change exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/365424

Change 365424 merged by Herron:
[operations/puppet@production] Lists: Change exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/365424

This issue should be resolved with Change https://gerrit.wikimedia.org/r/365424. However, since the filter that's been put in place is dependent on the sender address containing @qq.com and matching a regular expression it may require some care and feeding over time.

Huge thank you to Danny_B!

And to close the loop between email thread and phab:

On Jul 14, 2017, at 7:40 PM, Danny B. <Wikipedia.Danny.B@email.cz> wrote:

Hi,

just a quick status update after about a day of investigations and work on it:

• At about 16:30 UTC we have lowered down the spam score neccessary for the mail to be rejected. This has decreased the number of spams passing through to about one third. So at least some progress for >the time being. We couldn't decrease the threshold further though as it could affect some regular emails.
• We were testing various further filterings via matching of various things (since some used checkpoints in subjects varied during the time), because we didn't want to cut off the entire @qq.com. So the >frequency of spams randomly oscillated.
• At about 23:00 UTC we have tested the solution which seemed to finally catch everything properly as desired. So now there should be no such spams passing through, however, if senders will change the >structure of the subject, mails will pop up again, although - due to the first point - still in lower frequency.
• Should that happen, we are considering rejecting all mails from @qq.com (or <numbers>@qq.com) sent to any *-owner@lists.wikimedia.org address for the time being until narrower solution is set.

(We == Keith & myself.)

Please let us know if you're experiencing any further spams of this type since now.

Change 367677 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Lists: Change exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/367677

Change 367677 merged by Herron:
[operations/puppet@production] Lists: Change exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/367677

For the records, this seems to happen again targeting cep-owner@. (GMail puts them into the spam filter so I don't care too much.)