Page MenuHomePhabricator
Create Task
Maniphest T120210

tools-mail: check SPF of sender before forwarding email
Closed, ResolvedPublic
Actions

Description

I got two e-mails, headers:

Return-Path: <no-reply@tools.wmflabs.org>
Received: from mail.tools.wmflabs.org ([208.80.155.162]) by ***
 (***) with ESMTPS (Nemesis) id 0Lj04k-1ahbTk48ha-00dGID for
 <***>; Thu, 03 Dec 2015 10:06:39 +0100
Received: from [112.79.35.108/27.3.192.172]
        by mail.tools.wmflabs.org with esmtp (Exim 4.76)
        (envelope-from <no-reply@tools.wmflabs.org>)
        id 1a4PqU-0004C2-DR
        for gifti@tools.wmflabs.org; Thu, 03 Dec 2015 09:06:36/09:36:44 +0000
From: <no-reply@tools.wmflabs.org>
To: <gifti@tools.wmflabs.org>
Subject: Scanned image from MX-2600N
Date: Thu, 03 Dec 2015 14:36:20 +0530
Message-ID: <201512036317/201512034325.NOREPLY@tools.wmflabs.org>
MIME-Version: 1.0
Content-Type: multipart/mixed;
        boundary="----=_NextPart_000_7054_01D06642.8A15BEE0"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: Ac9zfUYGCX85GfHLR2eFV/vn3hBULA==

content:

Reply to: no-reply@tools.wmflabs.org> <no-reply@tools.wmflabs.org>>
Device Name: Not Set
Device Model: MX-2600N
Location: Not Set

File Format: DOC MMR(G4)
Resolution: 200dpi x 200dpi

Attached file is scanned image in DOC format.
Use Microsoft(R)Word(R) of Microsoft Systems Incorporated
to view the document.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+46 -1toolforge: mailrelay: introduce spam filtering with spamassassin
Customize query in gerrit

Related Objects
Search...

Event Timeline

Giftpflanze created this task.Dec 3 2015, 12:38 PM
Giftpflanze raised the priority of this task from to Needs Triage.
Giftpflanze updated the task description. (Show Details)
Giftpflanze added a project: Toolforge.
Giftpflanze added a subscriber: Giftpflanze.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptDec 3 2015, 12:38 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
valhallasw added a subscriber: valhallasw.Dec 3 2015, 12:45 PM

This e-mail indeed passed through the tool labs mailserver. In principle, this is OK (we are supposed to forward mails to gifti@tools.wmflabs.org to you), but we should obviously not accept mails 'from: <something>@tools.wmflabs.org' from a third party. Apparently we don't check SPF records ourselves...? :/

valhallasw added a comment.Dec 3 2015, 12:47 PM

see https://github.com/Exim/exim/wiki/SPF and /etc/exim4/conf.d/acl/30_exim4-config_check_rcpt

• chasemp triaged this task as Low priority.Dec 3 2015, 2:21 PM
• chasemp set Security to None.
coren added a subscriber: coren.Dec 3 2015, 2:44 PM

It's not as clearcut as that - you'd expect people receiving email to their *@tools.wmflabs.org addresses to be able to reply with that same address as From:; so it's not immediately clear that such emails should "obviously" not be accepted.

Email headers are not authenticated and cannot be; the best one can do is attempt to reduce obvious outliers and minimize spam and phishing and while using SPF to score spam is good, using it to reject email entirely is rarely a very good idea (as yahoo demonstrates with its horrid false positive rejections).

scfc added a subscriber: scfc.Dec 3 2015, 3:20 PM

I think people using tools.wmflabs.org as their domain for outgoing mails should use the proper mail server (which is tricky, but doable if you really want to), so I don't have any problem with publishing (hard) SPF records for tools.wmflabs.org.

But for incoming mails the situation is made worse by us currently forwarding mails without address rewriting, i. e. a mail from someone@somewhere to scfc@tools.wmflabs.org causes the Tools mail server to claim to my mail server that it is the legitimate sender for somewhere. So as long as we are not conservative in sending mails out, we should be liberal in accepting mails, not least so that we don't shoot ourselves in the foot :-).

valhallasw added a comment.EditedDec 3 2015, 3:26 PM

you'd expect people receiving email to their *@tools.wmflabs.org addresses to be able to reply with that same address as From

that doesn't work, actually, because we publish an SPF record:

  • from: valhallasw@gmail.com to marc@tools.wmflabs.org via smtp.gmail.com works (smtp.gmail.com is allowed to send mails from gmail.com)
  • from: valhallasw@tools.wmflabs.org to marc@tools.wmflabs.org via smtp.gmail.com works (tools.wmflabs.org doesn't check SPF, so accepts smtp.gmail.com as source)
  • from: valhallasw@tools.wmflabs.org to marc@otherdomain.org via smtp.gmail.com fails (smtp.gmail.com is not allowed to send mails from tools.wmflabs.org)
  • from: valhallasw@tools.wmflabs.org to marc@otherdomain.org via mail.tools.wmflabs.org fails (relay denied)

while using SPF to score spam is good, using it to reject email entirely is rarely a very good idea (as yahoo demonstrates with its horrid false positive rejections).

Iirc the issues with Yahoo are related to DMARC, not to SPF. We should add an envelope-from to our emails, though: T120225: Toolforge: correctly envelope forwarded email

valhallasw added a project: Mail.Dec 22 2015, 9:32 PM
valhallasw renamed this task from Weird e-mails from tool labs to tools-mail: check SPF of sender before forwarding email.May 27 2016, 12:22 PM
valhallasw moved this task from Triage to Backlog on the Toolforge board.
bd808 mentioned this in T249114: E-mails from noreply@pypi.org to tools.pywikibot@tools.wmflabs.org are not forwarded to certain recipients due to SPF.Apr 1 2020, 9:23 PM
bd808 mentioned this in T249237: Fix Cloud VPS and Toolforge mail servers to work with the modern internet.Apr 2 2020, 4:36 PM
bd808 added a parent task: T249237: Fix Cloud VPS and Toolforge mail servers to work with the modern internet.Apr 2 2020, 7:53 PM
herron added a subscriber: herron.May 4 2020, 10:35 PM

SPF checks/filtering can take some tuning, for a few reasons...

  • A SPF pass is essentially meaningless as anyone can register a domain and set a valid SPF record.
  • SPF "failure" can have multiple degrees set by the domain owner, which means...
  • SPF is often not going to be enough to trust or reject mail outright, but..
  • We could add different spam scores to messages depending on the type of failure.

What I think makes sense in this use case is:

  1. Roll out spamassassin to tools mail. We should be able to reuse the spamassassin puppet module and borrow config snippets from the mx exim template to get started (modules/role/templates/exim/exim4.conf.mx.erb)
  2. Log spam scores at first, then after some analysis start rejecting “high” spam scores (we drop spam-score >12 at the prod MXes, but maybe tools mail wants to be more aggressive)
  3. Set SPF hard failures (-all) to add a spamassassin spam score greater than that required to reject the mail outright during the smtp session

Step 1 would accomplish the task of "checking spf before forwarding" by itself, and steps 2-3 would go further towards better filtering of spam generally and preventing (ab)users from forging tools.wmflabs.org envelope from addresses (as tools.wmflabs.org has a hard fail SPF policy already).

aborrero claimed this task.Jun 25 2020, 4:06 PM
aborrero raised the priority of this task from Low to High.
aborrero edited projects, added cloud-services-team (Kanban); removed Cloud-Services.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
gerritbot added a comment.Jun 26 2020, 10:39 AM

Change 608005 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: mailrelay: introduce spam filtering with spamassassin

https://gerrit.wikimedia.org/r/608005

gerritbot added a project: Patch-For-Review.Jun 26 2020, 10:39 AM
Stashbot added a comment.Jun 26 2020, 12:12 PM

Mentioned in SAL (#wikimedia-cloud) [2020-06-26T12:12:32Z] <arturo> puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 (T120210)

aborrero added a comment.Jun 29 2020, 11:59 AM

I tested the new spam filter using GTUBE: https://spamassassin.apache.org/gtube/

Basically, sending an email with the mentioned string in the body: XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34@ (replace trailing @ with X).

A message with that string scores high spam value and is therefore rejected:

==> /var/log/exim4/rejectlog <==
2020-06-29 11:55:25 1jpsNh-0006AJ-BT H=mail-wr1-f54.google.com [209.85.221.54] X=TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<arturo.borrero.glez@gmail.com> rejected after DATA: This message scored 1000.0 spam points.
gerritbot added a comment.Jun 29 2020, 12:00 PM

Change 608005 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: mailrelay: introduce spam filtering with spamassassin

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608005

Stashbot added a comment.Jun 29 2020, 12:01 PM

Mentioned in SAL (#wikimedia-cloud) [2020-06-29T12:01:47Z] <arturo> introduced spam filter in the mail server (T120210)

aborrero lowered the priority of this task from High to Medium.Jul 1 2020, 11:40 AM

So, I've been working and testing the new antispam setup with @herron.

The setup we just introduced, as currently configured, correctly marks emails with SPF failures as possible spam. Therefore, such emails are likely going to hit end-user spam folder.
We could let exim drop all emails with SPF failures, by either increasing the SPF failure score or reducing general the general spam score required to drop a message. But at this point, that feels a little bit too much aggressive.

Instead, I suggest we leave this as is currently configured: marking SPF failures as possible spam messages. It is a significant improvement to what we had before anyways.
And in the future, if we get more reports related to this, we can review the setup again.

herron added a comment.Jul 1 2020, 5:20 PM
In T120210#6270976, @aborrero wrote:

Instead, I suggest we leave this as is currently configured: marking SPF failures as possible spam messages. It is a significant improvement to what we had before anyways.
And in the future, if we get more reports related to this, we can review the setup again.

Sounds reasonable to me! One additional action I'd suggest is reviewing the spamd logs in a months time to see how many mails are being flagged with SPF_FAIL, and review/tune based on that.

aborrero closed this task as Resolved.Jul 2 2020, 11:52 AM

Closing task now. Please anyone feel free to reopen if required.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL