Page MenuHomePhabricator

Massive spam to -owner mailing lists from *@qq.com emails
Closed, ResolvedPublic

Description

For some reason getting large amounts of spam from *@qq.com emails to checkuser-l-owner@lists.wikimedia.org (apparently roughly 10/min). Email from @Risker follows...

They're all coming in from email addresses like 396960395@qq.com (i.e., domain is the same, the numeric email account changes). Header info follows:

Delivered-To: risker.wp@gmail.com
Received: by 10.83.42.65 with SMTP id q62csp2462689yxq;
        Thu, 13 Jul 2017 09:13:39 -0700 (PDT)
X-Received: by 10.55.65.22 with SMTP id o22mr5463694qka.213.1499962419580;
        Thu, 13 Jul 2017 09:13:39 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1499962419; cv=none;
        d=google.com; s=arc-20160816;
        b=fDQMsD5JS/cd8gi/Ed+lDHMOz1ugi3JgkFdeTEAGwKC79hEUrKO7l4fRMBywtNBrCQ
         HeJqkJVltuQmrjk7h0gReA6IlwS63kr4ZRPFiURHgqMiamWgOwODJRKL8/GSUEkgWrtk
         OVUx/O8QSQtIWAj74GeMv1OWR1LvIaSltr6yAKG0zTOkuzwt/xZhYN26SEL2BRa4rz70
         VR8Bheojcsx1tCRxSUyS4zqcPTeiN5g6cz22vPRlbBP6fuMrc3ZmOxgC0bkNT6wLTouA
         3HJgdHoZWnSO7JHnQUiHf5CArHSVy2HxJvt57nc7Crd0AiB+/9Uh9ycnUjUeKeO5IuIJ
         P5rw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=sender:errors-to:from:to:subject:message-id:date:mime-version
         :dkim-signature:arc-authentication-results;
        bh=/E7avsRBSAWVcp2hQFRchBT/Jgzf7VfHLsoVX3rfUKU=;
        b=VqK+lBwvP6e65nRYQDA1KB7P/H65C00OVPTmcEcJj4eDtMEyHYrPCiQdjf7ZJP4GMf
         zTNG8HwfmrG8yythGsnacApM2EHtS2F4YxmqmyrYsLjZR7uCQXMh84t/Jt9ml3wGGuQx
         m1PrwkiR88f2dnhSYzFNJ6P99+y3M0Dh7g8Fmsds97lBdgTxlQ034TFgm26utK0zwPN6
         lslv1VL1dgCALaKy1DyVtDXyiMk2020YN5tByTOIUBwR03jaebGAvuD+mnVA1LV927J7
         uGsteCfAAcMfM2qmcHTyvBy7XPP8EhGLA77xhZhJtvamiJ6QeVBcNEN5Q+K0SnV5JPx8
         AeFw==
ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@lists.wikimedia.org header.b=sQToPzEZ;
       spf=pass (google.com: domain of mailman-bounces@lists.wikimedia.org designates 208.80.154.21 as permitted sender) smtp.mailfrom=mailman-bounces@lists.wikimedia.org;
       dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=qq.com
Return-Path: <mailman-bounces@lists.wikimedia.org>
Received: from lists.wikimedia.org (lists.wikimedia.org. [208.80.154.21])
        by mx.google.com with ESMTPS id q67si5555342qki.120.2017.07.13.09.13.39
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Thu, 13 Jul 2017 09:13:39 -0700 (PDT)
Received-SPF: pass (google.com: domain of mailman-bounces@lists.wikimedia.org designates 208.80.154.21 as permitted sender) client-ip=208.80.154.21;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@lists.wikimedia.org header.b=sQToPzEZ;
       spf=pass (google.com: domain of mailman-bounces@lists.wikimedia.org designates 208.80.154.21 as permitted sender) smtp.mailfrom=mailman-bounces@lists.wikimedia.org;
       dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=qq.com
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.wikimedia.org; s=wikimedia; h=Sender:Content-Type:From:To:Subject:Message-ID:Date:MIME-Version; bh=/E7avsRBSAWVcp2hQFRchBT/Jgzf7VfHLsoVX3rfUKU=; b=sQToPzEZv8oZ2GYr8kwRuX77xqXtSouEpQRzYTYFzNjRkuSqdF+cJ/LuKQ7HzJrKFI0aIA021Foo3IsDOI52uTV5q7+tuzRCgx2U4iOtPoLBPL0wFXLj0TCCnFDmAAPifxvKgp/8JG2bhFDnSYx15RvvnEylCVhizhbusbo5p3A=;
Received: from localhost ([::1]:46820 helo=fermium.wikimedia.org) by fermium.wikimedia.org with esmtp (Exim 4.84_2) (envelope-from <mailman-bounces@lists.wikimedia.org>) id 1dVgkE-00075e-Sg; Thu, 13 Jul 2017 16:13:39 +0000
Received: from [218.90.80.50] (port=58475 helo=helichina.com) by fermium.wikimedia.org with esmtp (Exim 4.84_2) (envelope-from <799808248@qq.com>) id 1dVgkC-00074s-9O for checkuser-l-owner@lists.wikimedia.org; Thu, 13 Jul 2017 16:13:37 +0000
MIME-Version: 1.0
Date: Fri, 14 Jul 2017 00:13:30 +0800
Message-ID: <8b0c233a717b3af2@bed15bbada91ee95>
Subject: 葛娜求-墺門永利331458。COM邀您住冊嶺5⒏赢58O提现, Jia客服扣:295019338瓴,只要投紸红宝天天抢,下鉒6和就送$特马最高51.8倍出口服务器过密
To: checkuser-l-owner@lists.wikimedia.org
Received: from helichina.com (unknown (53.78.60.206]) by helichina.com with SMTP id df0bed33-b417-4933-914b-b4fbd97df05a; for <checkuser-l-owner@lists.wikimedia.org>;Fri, 14 Jul 2017 00:13:30 +08:00
From: "关安" <799808248@qq.com>
Content-Type: multipart/alternative; boundary="074fd469-d9b2-484b-8eec-76dcf4541374"
Errors-To: mailman-bounces@lists.wikimedia.org
Sender: CheckUser-l <mailman-bounces@lists.wikimedia.org>
X-Spam-Score: 7.6 (+++++++)
X-Spam-Report: Spam detection software, running on the system "fermium.wikimedia.org", has identified this incoming email as possible spam.
  The original message has been attached to this so you can view it or label similar future email.
  If you have any questions, see the administrator of that system for details.
  Content preview:
  幻苍崖云æ ‘ 诗道有法,昔人贵在妙悟。……如禅门之作三观,如玄门之炼九还,观熟斯现心�
     , 李佳顺lijiashun [...]
   Content analysis details:
   (7.6 points, 4.0 required)
   pts rule name
              description ---- ---------------------- --------------------------------------------------
  0.0 BAD_ENC_HEADER
         Message has bad MIME encoding in the header
  0.2 FREEMAIL_ENVFROM_END_DIGIT Envelope-from freemail username ends in
                             digit (799808248[at]qq.com)
  0.0 FREEMAIL_FROM
          Sender email is commonly abused enduser mail provider
                             (799808248[at]qq.com)
  0.7 MPART_ALT_DIFF
         BODY: HTML and text parts are different
  0.0 HTML_MESSAGE
           BODY: HTML included in message
  1.1 MIME_HTML_ONLY
         BODY: Message only has text/html MIME parts
  0.0 RCVD_IN_SORBS_DUL
      RBL: SORBS: sent directly from dynamic IP address
                             [218.90.80.50 listed in dnsbl.sorbs.net]
  3.6 RCVD_IN_PBL
            RBL: Received via a relay in Spamhaus PBL
                             [218.90.80.50 listed in zen.spamhaus.org]
  0.0 UNPARSEABLE_RELAY
      Informational: message has unparseable relay lines
  0.0 MIME_HTML_ONLY_MULTI
   Multipart message only has text/html MIME parts
  0.6 HTML_MIME_NO_HTML_TAG
  HTML-only message, but there is no HTML tag
  0.0 MSGID_FROM_MTA_HEADER
  Message-Id was added by a relay
  1.3 RDNS_NONE
              Delivered to internal network by a host with no rDNS

Event Timeline

jrbs created this task.Jul 13 2017, 4:41 PM
jrbs created this object with visibility "Custom Policy".
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 13 2017, 4:41 PM
jrbs renamed this task from Massive spam to checkuser-l mailing list to Massive spam to checkuser-l-owner mailing list.Jul 13 2017, 4:42 PM
jrbs updated the task description. (Show Details)
jrbs renamed this task from Massive spam to checkuser-l-owner mailing list to Massive spam to -owner mailing lists.Jul 13 2017, 5:23 PM
jrbs triaged this task as Unbreak Now! priority.

Upping to UBN since this is affecting a lot of MLs, including now wikials-l-owner@.

Just to provide more complete information - approximately 1200 emails were received from the *@qq.com email addresses between 1330 UTC and 1634 UTC (when most of the list admins were removed as a stopgap measure, to stop the flooding of our personal email inboxes). They started at a rate of 1-2/minute and by the time I stopped receiving, they were coming at a rate of 15/minute. As best I can tell, this is the only *-L-owner mailing list affected - I am also a listadmin on a bunch of other lists and none of them were affected.

I have retained these mails temporarily, but plan to delete them on July 15 (they take up a lot of space, even if they're in the bin/spam folders), so if you need more examples or more information, please let me know soon.

jrbs added a subscriber: herron.Jul 13 2017, 5:32 PM

Further note - being reported on other -owner lists via the List Admins mailing list.

jrbs changed the visibility from "Custom Policy" to "All Users".Jul 13 2017, 5:56 PM
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptJul 13 2017, 5:56 PM

As a listadmin of meta-oversight, I confirm that I'm also receiving frequent spam to the list and the -owner address; although from different domains. Happy to share the domains if needed, although I clean my spam folder very regularly.

Recently in T161082 a default_bounce_matching_headers setting was deployed to hold messages with a 6 or greater spam score. I think this default is overridden when a list specific bounce_matching_headers setting is present (as is the case with checkuser-l and wikials-l). I went ahead and added these lines to the wikials-l bounce_matching_headers regexp:

from: .*@qq.com
X-Spam-Score:[^+]*[+]{6,}

For other lists IMHO it would be worth either a) clearing the lists bounce_matching_headers list override to inherit the default, or b) add to bounce_matching_headers something like this:

X-Spam-Score:[^+]*[+]{6,}
Framawiki added a project: Operations.EditedJul 13 2017, 6:46 PM

From https://wikitech.wikimedia.org/wiki/Mailman

Spam scores
The mailman UI supports this via the configuration variable header_filter_rules aka. 'Spam Filter Regexp' (description: Filter rules to match against the headers of a message.). See also https://www.gnu.org/software/mailman/mailman-admin/sender-filters.html
This can be found in the administrative interface in Privacy options...-> [Spam filters] -> Spam Filter Regexp (or visit directly the URL, replacing YOURLIST with your list name: https://lists.wikimedia.org/mailman/admin/YOURLIST/?VARHELP=privacy/spam/header_filter_rules ).

So someone from Operations has to check Spam Filter Regexp for each list ? Write a small script ? Directly from the database ?

From https://wikitech.wikimedia.org/wiki/Mailman

Spam scores
The mailman UI supports this via the configuration variable header_filter_rules aka. 'Spam Filter Regexp' (description: Filter rules to match against the headers of a message.). See also https://www.gnu.org/software/mailman/mailman-admin/sender-filters.html
This can be found in the administrative interface in Privacy options...-> [Spam filters] -> Spam Filter Regexp (or visit directly the URL, replacing YOURLIST with your list name: https://lists.wikimedia.org/mailman/admin/YOURLIST/?VARHELP=privacy/spam/header_filter_rules ).

So someone from Operations has to check Spam Filter Regexp for each list ? Write a small script ?

Someone with a better core technical understanding of mailman may correct me but my understanding is that while this would help for the mailing lists (and so may be useful) one of the other big issues here (the spam being sent to list OWNERS) would not be resolved. I don't think Mailman has any spam setting options for -owner addresses.

Perhaps we should lower the spam score that results in a deny at the lists MTA from 12 to 6. It would require a change in only one place and act as a default for lists addresses. List admins could still optionally set up additional filters through mailman admin.

So someone from Operations has to check Spam Filter Regexp for each list ? Write a small script ? Directly from the database ?

I don't see why "someone from Operations" has to do that hence removing tag.

On Jul 14, 2017, at 2:34 AM, Manuel Schneider <manuel.schneider@wikimedia.ch> wrote:
sorry, but I received another 600 such mails over night...
I think the point is that you added a filter to the list but these mails
should be bounced by the MTA right away. In this case the mails aren't
directed at the list but at the -owner alias, so I think Mailman filters
are not effective here.
/Manuel

KTC added a subscriber: KTC.Jul 14 2017, 3:53 PM
Legoktm changed the visibility from "All Users" to "Public (No Login Required)".Jul 14 2017, 4:08 PM

Change 365267 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Change lists to reject spam score of 6 or greater via exim acl

https://gerrit.wikimedia.org/r/365267

Change 365267 merged by Herron:
[operations/puppet@production] Change lists to reject spam score of 6 or greater via exim acl

https://gerrit.wikimedia.org/r/365267

Mentioned in SAL (#wikimedia-operations) [2017-07-14T16:36:46Z] <herron> lowered mailman/lists spam_score exim acl to 6 - T170601

herron added a subscriber: RobH.Jul 14 2017, 5:02 PM

365267 has been merged and I'm seeing loads of 'rejected after DATA' events for qq.com addresses in the exim logs now.

Change 365279 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Lists: Add exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/365279

Change 365279 merged by Herron:
[operations/puppet@production] Lists: Add exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/365279

Mentioned in SAL (#wikimedia-operations) [2017-07-14T18:54:16Z] <herron> added exim from/subject filter for spam observed from qq.com - T170601

Change 365424 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Lists: Change exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/365424

Change 365424 merged by Herron:
[operations/puppet@production] Lists: Change exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/365424

This issue should be resolved with Change https://gerrit.wikimedia.org/r/365424. However, since the filter that's been put in place is dependent on the sender address containing @qq.com and matching a regular expression it may require some care and feeding over time.

Huge thank you to Danny_B!

And to close the loop between email thread and phab:

On Jul 14, 2017, at 7:40 PM, Danny B. <Wikipedia.Danny.B@email.cz> wrote:
Hi,
just a quick status update after about a day of investigations and work on it:

  • At about 16:30 UTC we have lowered down the spam score neccessary for the mail to be rejected. This has decreased the number of spams passing through to about one third. So at least some progress for >the time being. We couldn't decrease the threshold further though as it could affect some regular emails.
  • We were testing various further filterings via matching of various things (since some used checkpoints in subjects varied during the time), because we didn't want to cut off the entire @qq.com. So the >frequency of spams randomly oscillated.
  • At about 23:00 UTC we have tested the solution which seemed to finally catch everything properly as desired. So now there should be no such spams passing through, however, if senders will change the >structure of the subject, mails will pop up again, although - due to the first point - still in lower frequency.
  • Should that happen, we are considering rejecting all mails from @qq.com (or <numbers>@qq.com) sent to any *-owner@lists.wikimedia.org address for the time being until narrower solution is set.

(We == Keith & myself.)
Please let us know if you're experiencing any further spams of this type since now.

herron lowered the priority of this task from Unbreak Now! to Normal.Jul 15 2017, 12:50 AM
herron removed a project: Patch-For-Review.
Aklapper renamed this task from Massive spam to -owner mailing lists to Massive spam to -owner mailing lists from *@qq.com emails.Jul 20 2017, 9:42 PM

Change 367677 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] Lists: Change exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/367677

Change 367677 merged by Herron:
[operations/puppet@production] Lists: Change exim filter for spam observed from qq.com

https://gerrit.wikimedia.org/r/367677

herron closed this task as Resolved.Jul 31 2017, 2:18 PM
herron removed a project: Patch-For-Review.

For the records, this seems to happen again targeting cep-owner@. (GMail puts them into the spam filter so I don't care too much.)