Hi @KTC, thanks for taking the time to report this!

See T66818: Mitigate strict DMARC policy on the mailing lists basically (which is resolved). :)

If there are specific issues to investigate, more information is welcome (potentially after turning this task into a restricted security task).

According to the message headers, fermium.wikimedia.org received a message from 154.73.18.196 at 09:53:16 UTC 15 March 2017 for delivery to a long list of Wikimedia mailing lists specifying "Wikimedia-l" <ktc AT ktchan.info> as the sender email address.

Received: from localhost ([::1]:51778 helo=fermium.wikimedia.org)
    by fermium.wikimedia.org with esmtp (Exim 4.84_2)
    (envelope-from <wikimedia-l-bounces@lists.wikimedia.org>)
    id 1co5cN-0007Sj-Gn; Wed, 15 Mar 2017 09:53:19 +0000
Received: from [154.73.18.196] (port=35800 helo=niitd.pisem.net)
    by fermium.wikimedia.org with esmtp (Exim 4.84_2)
    (envelope-from <ktc@ktchan.info>)
    id 1co5cJ-0007Ld-3r; Wed, 15 Mar 2017 09:53:16 +0000
From: "Wikimedia-l" <ktc@ktchan.info>

The message was accepted for delivery to at least Wikimedia-l, Wikisource-l, Gendergap, and WikimediaUK-l. It was held for moderation on at lest 8 others for the reason "Message has implicit destination".

ktchan.info specify a SPF record of:

v=spf1 +a +mx +ip4:87.76.28.86 -all

It happened again today, with the target GLAM, Wikisource-l, and Wikiquote-l, plus at least 6 others implicitly.

Received: from localhost ([::1]:54738 helo=fermium.wikimedia.org)
    by fermium.wikimedia.org with esmtp (Exim 4.84_2)
    (envelope-from <glam-bounces@lists.wikimedia.org>)
    id 1csahB-00022u-Tm; Mon, 27 Mar 2017 19:52:54 +0000
Received: from [221.199.61.194] (port=28951 helo=fpwm.yandex.ru)
    by fermium.wikimedia.org with esmtp (Exim 4.84_2)
    (envelope-from <ktc@ktchan.info>)
     id 1csah8-0001xj-4V; Mon, 27 Mar 2017 19:52:51 +0000

Is there anything that could be done from WM's server end to stop this? I'll also accept suggestion for what I can do on my end.

Thanks

Am I right to guess that we don't do (strict or else) SPF checking while we definitely should? Exim can handle SPF just fine alone, as well as spamassassin.
It's also a bit weird that we let an email to go with the flow with 10+ spam points, but maybe there are hist[oe]rical reasons...

In T160529#3135437, @KTC wrote:

I'll also accept suggestion for what I can do on my end.

Dropping/autorejecting email with matching header
​X-Spam-Score: .+\+\+\+\+\+
(which is above spam scrote 5.00) probably helps a lot.

In T160529#3161835, @grin wrote:

Dropping/autorejecting email with matching header
​X-Spam-Score: .+\+\+\+\+\+
(which is above spam scrote 5.00) probably helps a lot.

That's not something someone in my position can do since the email never goes through the legitimate (i.e. SPF authorised) server. It goes straight to WMF's server who send it out to list members. AFAIK, from the list members email server point of view, any SPF check will pass since it's checking WMF's mailman server.

In T160529#3162334, @KTC wrote:

AFAIK, from the list members email server point of view, any SPF check will pass since it's checking WMF's mailman server.

Indeed, see example (from a gmail recipient address):

Received-SPF: pass (google.com: domain of wikiquote-l-bounces@lists.wikimedia.org designates 208.80.154.75 as permitted sender) client-ip=208.80.154.75;

Do we need to install spf-tools-perl and set CHECK_RCPT_SPF=true in https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/role/templates/exim/exim4.conf.mx.erb ?
https://wiki.debian.org/Exim#SPF_filtering

In T160529#3162334, @KTC wrote:

In T160529#3161835, @grin wrote:

Dropping/autorejecting email with matching header
​X-Spam-Score: .+\+\+\+\+\+
(which is above spam scrote 5.00) probably helps a lot.

That's not something someone in my position can do since the email never goes through the legitimate (i.e. SPF authorised) server. It goes straight to WMF's server who send it out to list members. AFAIK, from the list members email server point of view, any SPF check will pass since it's checking WMF's mailman server.

Mailman can filter on headers and act (hold, reject) on it.

In T160529#3164199, @Nemo_bis wrote:

In T160529#3162334, @KTC wrote:

AFAIK, from the list members email server point of view, any SPF check will pass since it's checking WMF's mailman server.

Indeed, see example (from a gmail recipient address):

Received-SPF: pass (google.com: domain of wikiquote-l-bounces@lists.wikimedia.org designates 208.80.154.75 as permitted sender) client-ip=208.80.154.75;

Do we need to install spf-tools-perl and set CHECK_RCPT_SPF=true in https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/role/templates/exim/exim4.conf.mx.erb ?
https://wiki.debian.org/Exim#SPF_filtering

It depends on the versions of the components and also on the load of the servers. The layman's SPF check fork an external program and for a really busy server you may not want to do it and may choose spamassassin to handle it since it can be on a (or indeed many) remote server. I fabricate mailservers for a living so I may be a useful (though a bit low availability) resource to use. (Admittedly I am lazy to even check where the mail architecture is described, but if anyone points me to the specific info I'd read it and offer my opinions. I'm using multiple SA with redis and postgres backends, multiple clamav backends and exim with various levels of rate limits and rejections in SMTP time.)

We should definitely reject emails failing SPF. Much less forward that to mailing lists
(forwarding to list owners might be acceptable, although I wouldn't recommend that, as 99.999% will be spam).

T160529 + T133191 would probably solve T127247, once deployed on all mail exchanges.

In T160529#3164796, @grin wrote:

It depends on the versions of the components and also on the load of the servers.

https://ganglia.wikimedia.org/latest/graph_all_periods.php?title=mail+delivery&vl=&x=&n=&hreg[]=.*&mreg[]=exim.%2B&gtype=line&glegend=show&aggregate=1
https://ganglia.wikimedia.org/latest/graph_all_periods.php?title=incoming+mail&vl=&x=&n=&hreg[]=.*&mreg[]=exim.*in&gtype=line&glegend=show&aggregate=1
https://ganglia.wikimedia.org/latest/graph_all_periods.php?title=exim%20servers%20load&vl=&x=&n=&hreg[]=(fermium|iridium|kypton|mendelevium|mx1001|mx2001|phab2001|ununpentium)&mreg[]=(cpu_idle|load_five)&gtype=line&glegend=show&aggregate=1

if anyone points me to the specific info I'd read it and offer my opinions.

The canonical location is https://wikitech.wikimedia.org/wiki/Mail but you may learn more from puppet (https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/?find=(mail|exim)). I've added some links at https://wikitech.wikimedia.org/wiki/Mail#Puppet_configuration to start with.

@Nemo_bis uh, these servers are basically idle. Any SPF checking may be okay, fork or otherwise.

Thanks for the links, I'll take to browse them. (I am not a puppet guy so that takes a bit longer, but its syntax seems plain enough.)
Some brief observation:

• demime=* has been obsoleted for a while, depends on exim version
• SARE (for spamassassin) has been (though unofficially) deprecated in 2009 and, quote: "The SARE rules are broken to the point of being harmful. Do not use them." Unfortunately I cannot offer a good alternative, but considering the wiki mentioned design principle "better let spam through than break things" it's probably oughtn't be used anymore. Disclaimer: This advice is not always followed by myself, depend on how anal I want the spam filtering to be.
• possibly all the TLS stuff has or will be replaced by letsencrypt and the wiki could be updated

@Platonides unfortunately you have referenced restricted tasks so I cannot even see what their topic is. I possibly don't need to know it, so it's just a sidenote. ;-)

And again.... https://lists.wikimedia.org/pipermail/wikimedia-l/2017-April/087294.html

This seems like a pretty dangerous spear phishing vector when used by a skilled attacker.

In T160529#3213374, @KTC wrote:

And x2 again today. This is getting ridiculous.

Hello @KTC, as setting priorities should always reflect reality, not just cause it, would you please explain that why you believe that someone is working on this task soon?

I have no idea if the same person is behind this, or it's just a bit of haphazard pointy trolling, but this seems far too easy to disrupt email lists with cross-posted spam:
https://lists.wikimedia.org/pipermail/gendergap/2017-April/006589.html
Example from today, directed at me.

In T160529#3164792, @grin wrote:

Mailman can filter on headers and act (hold, reject) on it.

There's an explanation of how to do this per-list here: https://wikitech.wikimedia.org/wiki/Mailman#Spam_scores -- and a discussion of why we haven't done this globally here: T58525

As a bit of a temporary solution, I've added a hold rule on wikimedia-l for X-Spam-Score:[^+]*[+]{4,}. cc other list admins @Austin, @Ijon , @Esh77 . We may need to increase that 4 to a higher number if the false positives are too high, or abandon hope if false positives are still too high when we reach 10.

Ugh, the problem is not false positives, but that rule is increasing the amount of spam seen by list moderators. Presumably, somewhere, previously spam was getting discarded. Now it is getting held.
Since activating the rule, 3 hours ago. 5 spam messages have been sent to the moderation queue. These are of a type that we never normally need to see (e.g. three from %@%.date and one from %@%.review, two domains I've just now learnt existed, and immediately wish didnt exist).

From my experience combating spam on wikinews-l, I believe those with spam score +4 or above are usually safe to discard. (List is otherwise low traffic, so I can't guarantee about low false positives.)

Spam filtering based on X-Spam-Score is best discussed on T161082 IMHO (we need a global solution).

Another example (this one is probably a compromised mailbox contact list, rather than archive scraping): P5430

Another one occurred just now on wikimania-l, gendergap and wikiquote-l, again targeting Katie ;-(

The rules I am using on wikimedia-l to prevent this are causing a lot of work for the list admins. Please can we get a proper solution soon.

This happened again today, this time targeting checkuser-l and another user (will not disclose username here but one thing in common is that this user also uses mail on their own server). It is somewhat disturbing to receive spam on a non-public mailing list

In T160529#3283666, @NickK wrote:

This happened again today, this time targeting checkuser-l and another user (will not disclose username here but one thing in common is that this user also uses mail on their own server). It is somewhat disturbing to receive spam on a non-public mailing list

And also on stewards-l, same user, different content. I support the opinion of NickK. It is important that this is resolved soon...

In T160529#3260174, @jayvdb wrote:

Please can we get a proper solution soon.

Would that be T160529#3164928 or what would be "a proper solution"? (Trying to clarify)

Hello there. Today, I found that some e-mails that arrived to the wikimedia-co list may be related to this issue. All of them come from the -owner addresses of other Wikimedia lists, and some of them even imply that other lists have received messages from the wikimedia-co list itself.

A couple examples:

> Received: from localhost ([::1]:37766 helo=fermium.wikimedia.org)
> 	by fermium.wikimedia.org with esmtp (Exim 4.84_2)
> 	(envelope-from <wikimedia-l-bounces@lists.wikimedia.org>)
> 	id 1dN86L-000752-56
> 	for wikimedia-co@lists.wikimedia.org; Tue, 20 Jun 2017 01:37:05 +0000
> Subject: =?utf-8?q?=5BWe=5D_NF-e_LQNIH793052_=28Evelyn_e_Financeiro_E_R_L?=
>  =?utf-8?q?tda=29?=
> From: wikimedia-l-owner@lists.wikimedia.org
> To: wikimedia-co@lists.wikimedia.org
> MIME-Version: 1.0
> Content-Type: multipart/mixed; boundary="===============5680826425501144704=="
> Message-ID: <mailman.1312088.1497922620.30487.wikimedia-l@lists.wikimedia.org>
> Date: Tue, 20 Jun 2017 01:37:00 +0000
> Precedence: bulk
> X-BeenThere: wikimedia-l@lists.wikimedia.org
> X-Mailman-Version: 2.1.18
> List-Id: Wikimedia Mailing List <wikimedia-l.lists.wikimedia.org>
> X-List-Administrivia: yes
> Errors-To: wikimedia-l-bounces@lists.wikimedia.org
> Sender: "Wikimedia-l" <wikimedia-l-bounces@lists.wikimedia.org>

That spam apparently comes from the wikimedia-l list. On the other hand, check this one out:

Received: from localhost ([::1]:38396 helo=fermium.wikimedia.org)
	by fermium.wikimedia.org with esmtp (Exim 4.84_2)
	(envelope-from <wikimedia-gh-bounces@lists.wikimedia.org>)
	id 1dN8JU-0001Fh-0F
	for wikimedia-co@lists.wikimedia.org; Tue, 20 Jun 2017 01:50:40 +0000
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Subject: =?utf-8?q?Your_message_to_Wikimedia-GH_awaits_moderator_approval?=
From: wikimedia-gh-owner@lists.wikimedia.org
To: wikimedia-co@lists.wikimedia.org
Message-ID: <mailman.1312146.1497923438.30487.wikimedia-gh@lists.wikimedia.org>
Date: Tue, 20 Jun 2017 01:50:38 +0000
Precedence: bulk
X-BeenThere: wikimedia-gh@lists.wikimedia.org
X-Mailman-Version: 2.1.18
List-Id: Wikimedia Ghana User Group <wikimedia-gh.lists.wikimedia.org>
X-List-Administrivia: yes
Errors-To: wikimedia-gh-bounces@lists.wikimedia.org
Sender: "Wikimedia-GH" <wikimedia-gh-bounces@lists.wikimedia.org

This one's content is:

Your mail to 'Wikimedia-GH' with the subject

    [We] NF-e LQNIH793052 (Evelyn e Financeiro E R Ltda)

Is being held until the list moderator can review it for approval. The reason it is being held: Post by non-member to a members-only list

This implies that the wikimedia-gh list received this spam from the wikimedia-co list.

So i guess someone is sending spam subject lines to wikimedia-gh, with a forged from address of wikimedia-co@lists.wikimedia.org, in order for the mailing list software to resend the spam in the form of a pending moderation message. That's a really cute trick.

The problem with spoofed email addresses is happening again, this time on the Education mailing list.

https://lists.wikimedia.org/pipermail/education/2018-September/002082.html

https://lists.wikimedia.org/pipermail/education/2018-September/002069.html

@Pine: Please provide the message headers of one of these messages - unfortunately the archive at https://lists.wikimedia.org/pipermail/education/2018-September.txt.gz does not include them.

For this email:

Received: from localhost ([::1]:54186 helo=fermium.wikimedia.org) by fermium.wikimedia.org with esmtp (Exim 4.84_2) (envelope-from <education-bounces@lists.wikimedia.org>) id 1g2USD-0001ZW-Gh; Wed, 19 Sep 2018 04:51:11 +0000
Received: from o1.2e.shared.sendgrid.net ([167.89.100.140]:22977) by fermium.wikimedia.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from <bounces+1797473-a07b-education=lists.wikimedia.org@em5263.binstala.com.mx>) id 1g2US8-0001Z1-L8 for education@lists.wikimedia.org; Wed, 19 Sep 2018 04:51:07 +0000
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed;
  d=binstala.com.mx; h=from:to:subject:mime-version:content-type;
  s=s1; bh=srF3XPZTs+4d/OD/F0UujpqQv04=; b=dECVVEpETIh/jlgWv0dYuPI 2mg+61fHaWFJmQsL4WvyI8s5susXQVBsT9TIE+AgfqsbU72rWca3U7Yc/IElnsXI QdyZZm+1R/xf14h4VDo2Jb/PZSrUM0oRCRnLllwfFu8xOcsXMDgB1pchyc/ciyR2 Vvd37FumFKsa0SbBeg0w=
Received: by filter0064p3las1.sendgrid.net with SMTP id filter0064p3las1-28761-5BA1D5B6-11 2018-09-19 04:51:02.439527314 +0000 UTC m=+76873.541454894
Received: from 10.4.5.65 (host-181-199-31-74.ecua.net.ec [181.199.31.74]) by ismtpd0027p1mdw1.sendgrid.net (SG) with ESMTP id 0UqQvUbASpiPai96B5-i2A for <education@lists.wikimedia.org>; Wed, 19 Sep 2018 04:51:01.864 +0000 (UTC)
Date: Wed, 19 Sep 2018 04:51:03 +0000 (UTC)
From: "MCANDREW Ewan <Ewan.McAndrew@ed.ac.uk>" <karla.carrillo@binstala.com.mx>
To: education@lists.wikimedia.org
Message-ID: <3685427063847420425.A043CF8458F335FF@lists.wikimedia.org>
MIME-Version: 1.0
X-SG-EID: g+GBLqw9MVVW4y5eJAYzxGVPCdgepljsk/nQuhGVG1fXk1kiDqdbhJ4SGG6at9ydsw6zzC4rFZlUSG vryQhN7y7UlC6Ey81fRkhVqHHRriqXwudHLrDwD/hYIDEqK/rydahm/50RmNPxNWxHoxUMCuEWi+qr bJNfkG4Kw25/n/bk4qjLkiFv0TOtf1GtVMLpI4WefNcI9XQX2QO8ZeSToSx7V4F4y1cftj0MsQuW0h kG7eAFsynd7nJK+hcHkjaLprLDTGD7gX91OTIslNyP2KSSH6sV1BmTkO8sOgmEckI=
X-Content-Filtered-By: Mailman/MimeDel 2.1.18
Subject: [Wikimedia Education] MCANDREW Ewan Invoice # 7646592
X-BeenThere: education@lists.wikimedia.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: Wikimedia Education <education.lists.wikimedia.org>
List-Unsubscribe: <https://lists.wikimedia.org/mailman/options/education>, <mailto:education-request@lists.wikimedia.org?subject=unsubscribe>
List-Archive: <https://lists.wikimedia.org/pipermail/education/>
List-Post: <mailto:education@lists.wikimedia.org>
List-Help: <mailto:education-request@lists.wikimedia.org?subject=help>
List-Subscribe: <https://lists.wikimedia.org/mailman/listinfo/education>, <mailto:education-request@lists.wikimedia.org?subject=subscribe>
Reply-To: Wikimedia Education <education@lists.wikimedia.org>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Errors-To: education-bounces@lists.wikimedia.org
Sender: Education <education-bounces@lists.wikimedia.org>

For this email:

Received: from localhost ([::1]:46934 helo=fermium.wikimedia.org) by fermium.wikimedia.org with esmtp (Exim 4.84_2) (envelope-from <education-bounces@lists.wikimedia.org>) id 1fzd7H-00040I-DD; Tue, 11 Sep 2018 07:29:43 +0000
Received: from lex.balt.net ([217.117.28.48]:53332 helo=pastas.balt.net) by fermium.wikimedia.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from <info@litwestlumber.com>) id 1fzd7C-0003zv-Il for education@lists.wikimedia.org; Tue, 11 Sep 2018 07:29:40 +0000
Received: from 10.4.30.7 (unknown [177.226.252.139]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: info@litwestlumber.com) by pastas.balt.net (Postfix) with ESMTPSA id 94CAC1EAFD for <education@lists.wikimedia.org>; Tue, 11 Sep 2018 10:29:33 +0300 (EEST)
Date: Tue, 11 Sep 2018 02:29:39 -0600
From: "MCANDREW Ewan <Ewan.McAndrew@ed.ac.uk>" <info@litwestlumber.com>
To: education@lists.wikimedia.org
Message-ID: <17689245402176419449.87FE76AAE23A1A47@lists.wikimedia.org>
MIME-Version: 1.0
X-Virus-Scanned: clamav-milter 0.99.4 at neptunas3.balt.net
X-Virus-Status: Clean
X-Content-Filtered-By: Mailman/MimeDel 2.1.18
Subject: [Wikimedia Education] MCANDREW Ewan Merchandise: receipt #6899
X-BeenThere: education@lists.wikimedia.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: Wikimedia Education <education.lists.wikimedia.org>
List-Unsubscribe: <https://lists.wikimedia.org/mailman/options/education>, <mailto:education-request@lists.wikimedia.org?subject=unsubscribe>
List-Archive: <https://lists.wikimedia.org/pipermail/education/>
List-Post: <mailto:education@lists.wikimedia.org>
List-Help: <mailto:education-request@lists.wikimedia.org?subject=help>
List-Subscribe: <https://lists.wikimedia.org/mailman/listinfo/education>, <mailto:education-request@lists.wikimedia.org?subject=subscribe>
Reply-To: Wikimedia Education <education@lists.wikimedia.org>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Errors-To: education-bounces@lists.wikimedia.org
Sender: Education <education-bounces@lists.wikimedia.org>

Huh.

So the from headers aren't being spoofed. Only the human readable name is misleading.

So if this is a list that only allows posting by members, and info@litwestlumber.com / karla.carrillo@binstala.com.mx are not members, then this would indicate some sort of bug in mailman i guess?

I guess the first question would be, are info@litwestlumber.com & karla.carrillo@binstala.com.mx listed as members of the education list?

Based on mailman-users posts like this and this, mailman accepts an email if any of the addresses returned by [[https://bazaar.launchpad.net/~mailman-coders/mailman/2.1/view/head:/Mailman/Message.py#L179|get_senders()]] matches the membership list. That includes (by default, configurable via SENDER_HEADERS) the From header, the SMTP enveloper sender, the Reply-To: header and the Sender: header. Is it possible that the email contained a whitelisted Reply-To: or Sender: address (which would not be visible by list members since mailman replaces those with its own address)?

I sent an email to the Education list admins to request that they comment in this ticket in response to the questions above.