Page MenuHomePhabricator

@txt.att.net bounce notifications being sent to list admins
Closed, InvalidPublic

Description

Hello Operations,

OIT received a ticket from Samir (admin for the education-collab list).
He's wondering why he's receiving the following message every couple of hours.

Please advise.

Eliza

////

Hi colleagues,

Do you have any idea what do these messages mean?

Thank you!

Samir Elsharbaty,
Wikipedia Education Program

Wikimedia Foundation

+2.011.200.696.77

education.wikimedia.org

  • Forwarded message ----------

From: <mailman-bounces@lists.wikimedia.org>
Date: Thu, Sep 17, 2015 at 6:01 AM
Subject: Uncaught bounce notification
To: education-collab-owner@lists.wikimedia.org

The attached message was received as a bounce, but either the bounce
format was not recognized, or no member addresses could be extracted
from it. This mailing list has been configured to send all
unrecognized bounce messages to the list administrator(s).

For more information see:
https://lists.wikimedia.org/mailman/admin/education-collab/bounce

  • Forwarded message ----------

From: postmaster@txt.att.net
To: education-collab-bounces@lists.wikimedia.org
Cc:
Date: Wed, 16 Sep 2015 23:01:04 -0400
Subject: Unable to deliver message.

This Message was undeliverable due to the following reason: the subscriber has restricted e-mail to <2524063603@mms.att.net> Please reply to <Postmaster@txt.att.com> if you feel this message to be in error.

  • Forwarded message ----------

From:
To:
Cc:
Date:
Subject:
X-Cloudmark-Analysis: v=2.1 cv=E5Ne+8tl c=1 sm=1 tr=0
a=QCBgvn5l3w1Vc5bg4KgrXw==:117 a=QCBgvn5l3w1Vc5bg4KgrXw==:17 a=3GbmggnxAAAA:8
a=IkcTkHD0fZMA:10 a=xqWC_Br6kY4A:10 a=ff-B7xzCdYMA:10 a=HZJGGiqLAAAA:8
a=NAi6eCUdRxSACJAc2A8A:9 a=QEXdDO2ut3YA:10 a=2tg8LeLMCKAA:10
Reply-To: <education-collab-request@lists.wikimedia.org>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.wikimedia.org; s=wikimedia;
h=Sender:List-Id:Date:Message-ID:Reply-To:Subject:To:From:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=kbhFMGjY5Ikc12Hf1DzOcP+S7kqAtBGDx6zge/s4HeE=;
b=QaQRZoPbgZrVei778vEyvGoz0PnxgdNGfE3AtL8+TB7/PvnYf6mOXAaJTJC3OenrDd8BJPJM2aD8CbU+68mc0wZYfw4l4h1x88sp/IjVwQ0EmDSnYzTASr1svmNbo/xHx3EU1LvqJDA0nu+Drol6Ru+GFYSJvt4Wo5Hi4eMA/90=;
Received: from localhost ([::1]:49415 helo=sodium.wikimedia.org)
by sodium.wikimedia.org with esmtp (Exim 4.71)
(envelope-from <education-collab-bounces@lists.wikimedia.org>)
id 1ZcPRX-0002QN-Kf
for 2524063603@mms.att.net; Thu, 17 Sep 2015 03:01:03 +0000
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
From: education-collab-request@lists.wikimedia.org
To: 2524063603@mms.att.net
Subject: confirm becbbf646e5eee5169c0ad58145fd999fca1498c
Reply-To: education-collab-request@lists.wikimedia.org
Message-ID: <mailman.0.1442458826.8475.education-collab@lists.wikimedia.org>
Date: Thu, 17 Sep 2015 03:00:26 +0000
Precedence: bulk
X-BeenThere: education-collab@lists.wikimedia.org
X-Mailman-Version: 2.1.13
List-Id: "The Education Collaborative \(Public List\)"
<education-collab.lists.wikimedia.org>
X-List-Administrivia: yes
Sender: education-collab-bounces@lists.wikimedia.org
Errors-To: education-collab-bounces@lists.wikimedia.org

TWFpbGluZyBsaXN0IHN1YnNjcmlwdGlvbiBjb25maXJtYXRpb24gbm90aWNlIGZvciBtYWlsaW5n
IGxpc3QKRWR1Y2F0aW9uLWNvbGxhYgoKV2UgaGF2ZSByZWNlaXZlZCBhIHJlcXVlc3QgZnJvbSAx
MTQuMTA5LjgxLjEyMSBmb3Igc3Vic2NyaXB0aW9uIG9mCnlvdXIgZW1haWwgYWRkcmVzcywgIjI1
MjQwNjM2MDNAbW1zLmF0dC5uZXQiLCB0byB0aGUKZWR1Y2F0aW9uLWNvbGxhYkBsaXN0cy53aWtp
bWVkaWEub3JnIG1haWxpbmcgbGlzdC4gIFRvIGNvbmZpcm0gdGhhdAp5b3Ugd2FudCB0byBiZSBh
ZGRlZCB0byB0aGlzIG1haWxpbmcgbGlzdCwgc2ltcGx5IHJlcGx5IHRvIHRoaXMKbWVzc2FnZSwg
a2VlcGluZyB0aGUgU3ViamVjdDogaGVhZGVyIGludGFjdC4gIE9yIHZpc2l0IHRoaXMgd2ViIHBh
Z2U6CgogICAgaHR0cHM6Ly9saXN0cy53aWtpbWVkaWEub3JnL21haWxtYW4vY29uZmlybS9lZHVj
YXRpb24tY29sbGFiL2JlY2JiZjY0NmU1ZWVlNTE2OWMwYWQ1ODE0NWZkOTk5ZmNhMTQ5OGMKCgpP
ciBpbmNsdWRlIHRoZSBmb2xsb3dpbmcgbGluZSAtLSBhbmQgb25seSB0aGUgZm9sbG93aW5nIGxp
bmUgLS0gaW4gYQptZXNzYWdlIHRvIGVkdWNhdGlvbi1jb2xsYWItcmVxdWVzdEBsaXN0cy53aWtp
bWVkaWEub3JnOgoKICAgIGNvbmZpcm0gYmVjYmJmNjQ2ZTVlZWU1MTY5YzBhZDU4MTQ1ZmQ5OTlm
Y2ExNDk4YwoKTm90ZSB0aGF0IHNpbXBseSBzZW5kaW5nIGEgYHJlcGx5JyB0byB0aGlzIG1lc3Nh
Z2Ugc2hvdWxkIHdvcmsgZnJvbQptb3N0IG1haWwgcmVhZGVycywgc2luY2UgdGhhdCB1c3VhbGx5
IGxlYXZlcyB0aGUgU3ViamVjdDogbGluZSBpbiB0aGUKcmlnaHQgZm9ybSAoYWRkaXRpb25hbCAi
UmU6IiB0ZXh0IGluIHRoZSBTdWJqZWN0OiBpcyBva2F5KS4KCklmIHlvdSBkbyBub3Qgd2lzaCB0
byBiZSBzdWJzY3JpYmVkIHRvIHRoaXMgbGlzdCwgcGxlYXNlIHNpbXBseQpkaXNyZWdhcmQgdGhp
cyBtZXNzYWdlLiAgSWYgeW91IHRoaW5rIHlvdSBhcmUgYmVpbmcgbWFsaWNpb3VzbHkKc3Vic2Ny
aWJlZCB0byB0aGUgbGlzdCwgb3IgaGF2ZSBhbnkgb3RoZXIgcXVlc3Rpb25zLCBzZW5kIHRoZW0g
dG8KZWR1Y2F0aW9uLWNvbGxhYi1vd25lckBsaXN0cy53aWtpbWVkaWEub3JnLgo=

  • Forwarded part --

Mailing list subscription confirmation notice for mailing list
Education-collab

We have received a request from 114.109.81.121 for subscription of
your email address, "2524063603@mms.att.net", to the
education-collab@lists.wikimedia.org mailing list. To confirm that
you want to be added to this mailing list, simply reply to this
message, keeping the Subject: header intact. Or visit this web page:

https://lists.wikimedia.org/mailman/confirm/education-collab/becbbf646e5eee5169c0ad58145fd999fca1498c

Or include the following line -- and only the following line -- in a
message to education-collab-request@lists.wikimedia.org:

confirm becbbf646e5eee5169c0ad58145fd999fca1498c

Note that simply sending a `reply' to this message should work from
most mail readers, since that usually leaves the Subject: line in the
right form (additional "Re:" text in the Subject: is okay).

If you do not wish to be subscribed to this list, please simply
disregard this message. If you think you are being maliciously
subscribed to the list, or have any other questions, send them to
education-collab-owner@lists.wikimedia.org.

unnamed_attachment_1.eml

Event Timeline

eliza created this task.Sep 17 2015, 4:27 PM
eliza raised the priority of this task from to Needs Triage.
eliza updated the task description. (Show Details)
eliza added a subscriber: eliza.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2015, 4:27 PM
Krenair added a subscriber: Krenair.

Adding operations

Restricted Application added a subscriber: Matanya. · View Herald TranscriptSep 17 2015, 4:29 PM

I think @MarkTraceur was seeing something similar?

I've been seeing these reports on multimedia-team and multimedia-alerts. None of the postmaster addresses work, and none of the phone number addresses work, either. I can't seem to find any setting to disable these notices, nor to block this e-mail address, in mailman.

Dzahn added a subscriber: Dzahn.Sep 17 2015, 5:39 PM

"If bounce_unrecognized_goes_to_list_owner is Yes, any message received at
the listname-bounces address which is not recognized by Mailman as a
DSN is forwarded to the list owner as an unrecognized bounce since at
that point, Mailman doesn't know if it was an actual undeliverable
mail DSN or something else."

https://mail.python.org/pipermail/mailman-users/2007-December/059531.html

Dzahn added a comment.Sep 17 2015, 5:46 PM

quote from mailman UI:

//While Mailman's bounce detector is fairly robust, it's impossible to detect every bounce format in the world. You should keep this variable set to Yes for two reasons: 1) If this really is a permanent bounce from one of your members, you should probably manually remove them from your list, and 2) you might want to send the message on to the Mailman developers so that this new format can be added to its known set.

If you really can't be bothered, then set this variable to No and all non-detected bounces will be discarded without further processing. //

eliza added a comment.Sep 17 2015, 6:05 PM

Thank you Dzahn,

So I should convey the following information to Samir?

Should Mailman perform automatic bounce processing? <
https://lists.wikimedia.org/mailman/admin/education-collab/?VARHELP=bounce/bounce_processing

Should Mailman send you, the list owner, any bounce messages that failed to
be detected by the bounce processor? Yes is recommended. <
https://lists.wikimedia.org/mailman/admin/education-collab/?VARHELP=bounce/bounce_unrecognized_goes_to_list_owner

and the general text above the other bounce settings <
https://lists.wikimedia.org/mailman/admin/education-collab/bounce>

These are per-list settings.

quote from mailman UI:

//While Mailman's bounce detector is fairly robust, it's impossible to
detect every bounce format in the world. You should keep this variable set
to Yes for two reasons: 1) If this really is a permanent bounce from one of
your members, you should probably manually remove them from your list, and

  1. you might want to send the message on to the Mailman developers so that

this new format can be added to its known set.

If you really can't be bothered, then set this variable to No and all
non-detected bounces will be discarded without further processing. //

The same thing is happening on the social-media, research-newsletter and wikimediaannounce-l lists.

eliza added a comment.Sep 17 2015, 8:59 PM

Hello All,

Not sure if I should send another Phabrictor ticket, but we just received
this ticket from Nick:

Hi! I'm getting a number of these messages (18 over the last 24 hours

sent to ee-owner@ and design-owner@ (2 of the lists I administer).
I've searched the list of subscribers, but there aren't any that use
an att.net address (I just ctrl-F searched for "att", on each of the
alphabetical pages of members).
Any idea what I should do, next?
Thanks!
Nick / Quiddity

Seems like the same situation.
Please advise.

Thank you,
Eliza

Barras added a subscriber: Barras.Sep 17 2015, 9:35 PM
Pine added a subscriber: Pine.Sep 17 2015, 9:35 PM

We're having this same problem on the Cascadia mailing list.

Krenair renamed this task from education-collab-owner@lists.wikimedia.org bounce notification to @txt.att.net bounce notifications being sent to list admins.Sep 17 2015, 9:39 PM
Krenair set Security to None.

Same for mediawiki-commits.

Same for checkuser-l, overight-wp-simple-l, cvn-l, publicpolicy and probably any other list I administer (I already deleted most mails)

Okay, I think it's clear now that this is happening on plenty of lists. :)

Tgr added a subscriber: Tgr.Sep 17 2015, 9:44 PM

Also fundraiser

AND MY AXE!

Wait, I mean, and my mailing lists. The Discovery Department's mailing lists are affected by this too.

JohnLewis triaged this task as Medium priority.Sep 17 2015, 10:13 PM
JohnLewis added a subscriber: JohnLewis.

As Krenair said, enough lists is enough. This ticket is likely generating more spam than these bounces together. I'll take a look at this tomorrow, likely post migration (as I'll be able to actually look then).

Thank you everyone !

I saw 12 of them in 24 hours for one of the lists I administer. All of them for the same number as in the above report, but note it was @mms.att.net, not txt.att.net (I have seen that in the past, though, it's something that happens from time to time). Each subscription request came from a different IP address, mostly Chinese (9) but also from Poland, Colombia and Brazil.

I don't really know what's the point in doing this (flood someone's phone?) Is there any reasonable need of receiving a mailing list at txt/mms.att.net? (would someone want to do that?) Otherwise, banning the domain for subscribing would be appropriate.

Ahonc added a subscriber: Ahonc.Sep 17 2015, 11:26 PM
Ocaasi added a subscriber: Ocaasi.Sep 18 2015, 12:04 AM

Just so I don't feel left out, Wikipedia Library list is getting one every few hours.

I don't really know what's the point in doing this (flood someone's phone?) Is there any reasonable need of receiving a mailing list at txt/mms.att.net? (would someone want to do that?) Otherwise, banning the domain for subscribing would be appropriate.

Honestly I'd be supportive of just banning the domain from a root level, especially if it can stop email to -owner too because the domain is one of the more common ones I've seen for spam emails in general. So far up to 50 of these bounce notifications ;)

Risker added a subscriber: Risker.Sep 18 2015, 2:05 AM

This has happened before, to varying extents. It strikes me that someone has found a way to spoof the WMF mailing list addresses, and the ATT group is sending bounces. We get a lot of bounces like this for the (very longstanding) checkuser list - not just from ATT - a few a week, at least.

Same for wikimedia-ped

Thanks to all who are working to solve this! :-)

Just so I don't feel left out, Wikipedia Library list is getting one every few hours.

I'm up to 30 of these "Uncaught bounce notification" e-mails. :-)

Has someone tried just calling the number and explaining the situation? (It seems to be a real US phone number with a North Carolina area code, with reverse search resolving to a real-sounding name, "R.B.". See also https://www.att.com/esupport/article.jsp?sid=KB92125&cv=820 )

If Platonides' hunch is right and this is a failed attempt to harass this person (failed because, see above, "the subscriber has restricted e-mail" to their number), they might be interested in the originating IPs. (Which BTW, @eliza, one might generally want to redact for privacy reasons before posting something like this on a public bug tracker - but in this case there's probably not much harm done, as that Thai IP appears to be one of many proxys used by an attacker.)

Dzahn added a comment.EditedSep 18 2015, 5:13 AM

So I should convey the following information to Samir?

@eliza Yes, please forward my original comment along with this:

This message is also for all other affected list admins:

You can go to the following URL, where you have to replace <LISTNAME> with the name of your list:

https://lists.wikimedia.org/mailman/admin/<LISTNAME>/?VARHELP=bounce/bounce_unrecognized_goes_to_list_owner

login with your admin password., and switch from Yes to No

"Should Mailman send you, the list owner, any bounce messages that failed to be detected by the bounce processor? Yes is recommended."

Then hit "Submit your changes", and let me know if it helped.

Also see the comments above though why mailman says this is not a good permanent solution to just ignore all of them. Legitimate bounces should still be checked and unsubscribed where needed.

Meno25 added a subscriber: Meno25.Sep 18 2015, 5:55 AM
Isarra added a subscriber: Isarra.Sep 18 2015, 6:40 AM
Melos added a subscriber: Melos.Sep 18 2015, 11:47 AM

This is getting annoying. Can someone just patch this up to get rid of this junk?

Multichill raised the priority of this task from Medium to High.Sep 18 2015, 1:38 PM
Dzahn added a comment.Sep 18 2015, 1:40 PM

@Multichill Did you try what i suggested in my comment right above?

@Multichill Did you try what i suggested in my comment right above?

I'm admin for quite a few lists. And as pointed out this is not advised in the manual. This is a server wide problem and should be solved server wide, not on a per lists basis.

JohnLewis lowered the priority of this task from High to Medium.

First comment for the sake of everyone: please can people stop commenting things that add no value or enhance the ability for us to look into the issue. the 'xx list also' comments are useless and just ironically generate more emails for people.

Now my comment: I'm assigning this to myself and I will look at it in detail when I can. This is extremely low priority right now considering the mailman migration and that follow up (security over mindless issues). This *will* be addressed through the correct channels. Per-list things are the only thing that can be done as mailman has no concept of global list configuration. since these are bounces by mailman, there is nothing we can do to solve that issue besides fixing *this* issue.

I hope everyone respects that our focus right now is elsewhere within mailman and that more comments don't really make things move faster (in fact we're discussing this so much, the migration is become more and more likely to take longer).

Thanks!

Hi John,

First comment for the sake of everyone: please can people stop commenting things that add no value or enhance the ability for us to look into the issue. the 'xx list also' comments are useless and just ironically generate more emails for people.
Now my comment: I'm assigning this to myself and I will look at it in detail when I can.

Last time I checked you were not on the operations team or do you have shell access to the machine?

This is extremely low priority right now considering the mailman migration and that follow up (security over mindless issues). This *will* be addressed through the correct channels. Per-list things are the only thing that can be done as mailman has no concept of global list configuration. since these are bounces by mailman, there is nothing we can do to solve that issue besides fixing *this* issue.

You're looking the wrong place. The mailman server runs Exim. In Exim you can just setup a blackhole route send this junk to /dev/null .

I hope everyone respects that our focus right now is elsewhere within mailman and that more comments don't really make things move faster (in fact we're discussing this so much, the migration is become more and more likely to take longer).

No, I don't agree with that. This is a classic IT problem: The current product is not done or malfunctioning and the new product isn't ready yet. I'd suggest you have an operation engineer with exim experience set the null route so this ticket can be closed and you have your hands free for the migration. I'm sure ops is more than willing to help out here.

Hi John,

First comment for the sake of everyone: please can people stop commenting things that add no value or enhance the ability for us to look into the issue. the 'xx list also' comments are useless and just ironically generate more emails for people.
Now my comment: I'm assigning this to myself and I will look at it in detail when I can.

Last time I checked you were not on the operations team or do you have shell access to the machine?

Yes. I have sudo access to the mailman server.

This is extremely low priority right now considering the mailman migration and that follow up (security over mindless issues). This *will* be addressed through the correct channels. Per-list things are the only thing that can be done as mailman has no concept of global list configuration. since these are bounces by mailman, there is nothing we can do to solve that issue besides fixing *this* issue.

You're looking the wrong place. The mailman server runs Exim. In Exim you can just setup a blackhole route send this junk to /dev/null .

What junk? Bounces are important to list administrators and also site administrators to help debug issues.

I hope everyone respects that our focus right now is elsewhere within mailman and that more comments don't really make things move faster (in fact we're discussing this so much, the migration is become more and more likely to take longer).

No, I don't agree with that. This is a classic IT problem: The current product is not done or malfunctioning and the new product isn't ready yet. I'd suggest you have an operation engineer with exim experience set the null route so this ticket can be closed and you have your hands free for the migration. I'm sure ops is more than willing to help out here.

Just to put it into perspective - the migration is *right now*. Likely to finish in less than hour.

revi added a subscriber: revi.Sep 18 2015, 3:26 PM
revi added a comment.Sep 18 2015, 3:32 PM

This is extremely low priority right now considering the mailman migration and that follow up (security over mindless issues). This *will* be addressed through the correct channels. Per-list things are the only thing that can be done as mailman has no concept of global list configuration. since these are bounces by mailman, there is nothing we can do to solve that issue besides fixing *this* issue.

You're looking the wrong place. The mailman server runs Exim. In Exim you can just setup a blackhole route send this junk to /dev/null .

What junk? Bounces are important to list administrators and also site administrators to help debug issues.

Obviously this att junk which is hitting our inbox. This bounce is not important, just annoying.

I'm going to agree with John here that bounces are important. They are flags that there is a problem with the process - the level of importance of the problem will vary. That lots of lists are getting the same flag repeatedly is actually quite useful information that helps to define and prioritize this issue.

I also agree that this issue should take lower priority than the conversion that had already been scheduled and is in the middle of implementation. Once the conversion is complete, I think it would be reasonable to expect that its priority will be escalated.

Thank you for taking care of it, Eliza and John!

Should Mailman send you, the list owner, any bounce messages that failed to be detected by the bounce processor? Yes is recommended.

@JohnLewis do you recommend changing it to "No" temporarily until the issue is solved?

Tgr added a comment.Sep 18 2015, 8:11 PM

You can probably just add 2524063603@mms.att.net to https://lists.wikimedia.org/mailman/admin/<listname>/?VARHELP=privacy/subscribing/ban_list if you are concerned about unsubscribing from bounces altogether. That said, the reason given in the mailman manual to not disable bounce reporting is that you can report suspicious bounces to the mailman developers if you see them. If you don't really see yourself doing that (especially for a mailman release that's fifteen months old), you shouldn't worry about disabling it IMO.

Dzahn added a comment.Sep 18 2015, 8:26 PM

Should Mailman send you, the list owner, any bounce messages that failed to be detected by the bounce processor? Yes is recommended.

@JohnLewis do you recommend changing it to "No" temporarily until the issue is solved?

Yes, just do that if it bothers you, as already suggested twice in this thread.

Quiddity removed a subscriber: Quiddity.Sep 18 2015, 9:29 PM
JohnLewis closed this task as Invalid.Sep 21 2015, 5:44 PM

It's subscription spam again (has happened in the past). Has anyone received any recently? My last bounce was September 18th and since then nothing else. Closing as invalid, re-open if this is still happening as looking at logs on fermium, this is not occurring.

Meno25 removed a subscriber: Meno25.Sep 30 2015, 11:55 AM