Page MenuHomePhabricator

mass Yahoo / AOL bounces mailman
Open, MediumPublic

Description

About 9 hours ago I got a massive number of disabled subscriptions (40-50) of aol.com addresses because of excessive bounces ~10h ago. Bounce notifications are mostly of the type:

SMTP error from remote mail server after MAIL FROM:<wikilovesmonuments-bounces@lists.wikimedia.org> SIZE=16180:

host mx-aol.mail.gm0.yahoodns.net [98.137.157.43]:
421 4.7.0 [TSS04] Messages from 208.80.154.21 temporarily deferred due to user complaints - 4.16.55.1; see https://help.yahoo.com/kb/postmaster/SLN3434.html:
retry timeout exceeded

Let me know if I can provide more detail. Mailing list in question is wikilovesmonuments but sounds like a more generic issue is at the core.

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptSep 9 2019, 11:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Effeietsanders renamed this task from mass AOL bounces on mailman to mass AOL bounces mailman.Sep 9 2019, 11:44 PM
Effeietsanders updated the task description. (Show Details)
Effeietsanders updated the task description. (Show Details)

Always helpful when Yahoo does not list that error code on Yahoo's page that is supposed to list their error codes...

jbond triaged this task as Medium priority.Sep 10 2019, 10:10 AM
jbond added a project: Mail.
jbond added a subscriber: jbond.
herron added a subscriber: herron.Sep 10 2019, 1:44 PM

These bounces are happening occasionally now.

Platonides added a subscriber: Platonides.EditedSep 22 2019, 8:11 PM

Yesterday, we had the same issue on biblio-es-l with all subscribers using a yahoo.es email address being automatically disabled delivery (funnily, it was not the case for those with a yahoo.com one), as the max retry timeout for emails from an email from Tuesday was reached.

Error received was the same:

 SMTP error from remote mail server after MAIL FROM:<XXXXXX-l-bounces@lists.wikimedia.org> SIZE=15609:
host mx-eu.mail.am0.yahoodns.net [188.125.72.73]:
421 4.7.0 [TSS04] Messages from 208.80.154.21 temporarily deferred due to user complaints - 4.16.55.1; see https://help.yahoo.com/kb/postmaster/SLN3434.html:
retry timeout exceeded

(208.80.154.21 is lists.wikimedia.org)

https://help.yahoo.com/kb/postmaster/SLN3434.html mentions:

  • The error is temporary. We encourage you to retry sending emails 4 hours after encountering the error.
  • If you're an administrator of message content and mailing policy and you've deployed significant changes or you've received this error for more than 48 hours, we ask that you review outgoing messages for objectionable content or practices.

IMHO this is likely a false positive on Yahoo! side considering that the emails from list are spam. Most likely, that will be caused by blocked/discarded emails sent to mailing list owners, since mailman will take care of everything on https://help.yahoo.com/kb/postmaster/recommended-guidelines-bulk-mail-senders-postmasters-sln3435.html However, it will damage the service for yahoo subscribers. I suppose there is a FBL with yahoo, that could be checked, too?

Platonides renamed this task from mass AOL bounces mailman to mass Yahoo / AOL bounces mailman.Sep 22 2019, 8:11 PM

See T22507 for the same issue in 2009

Similarly, on Wednesday ...@yahoo.com subscriptions were disabled for wikitech-l due to this same issue, bouncing https://lists.wikimedia.org/pipermail/wikitech-l/2019-September/092531.html, a legitimate email which seems perfectly innocuous.

Effeietsanders raised the priority of this task from Medium to High.Oct 8 2019, 3:18 PM

Now the subscriptions were not just disabled, but some 30+ were actually unsubscribed. We're doing a huge disservice to community members that have an AOL or Yahoo account by not following up on this.
Changing priority from normal to high.

An estimated 120 emails have now been unsubscribed. It looks like AOL and Yahoo. Is this also happening for other mailing lists?

Paladox added a subscriber: Paladox.EditedOct 11 2019, 5:10 PM

When i sent a email to wikitech-i last night, it failed (seems to be because yahoo blacklisted lists.wikimedia.org).

I'm wondering if this is somehow related to the massive spam attack we had a few months ago on some mailing-lists (hundred of fake AOL email addresses subscribed to the Wikidata mailing-list, and were bounced out a few weeks later)

Aklapper lowered the priority of this task from High to Medium.Oct 11 2019, 5:56 PM

An estimated 120 emails have now been unsubscribed. It looks like AOL and Yahoo. Is this also happening for other mailing lists?

Yes, same for wikitech-l@, and I consider that normal and expected, as those were [nearly] all spam accounts as explained by Lea in the last comment.

Is this normal though? Having lists.wikimedia.org blocked by yahoo in my opinion is pretty high priority.

aezell added a subscriber: aezell.Oct 11 2019, 6:08 PM

I spoke to a friend who still works in this area and they said that spam detection and management is in freefall at Yahoo/AOL right now. They are rapidly defunding that part of the business and many automated operations are happening with little human oversight.

This is obviously anecdotal and not actionable. I mention it more to say that there may be operational issues at those vendors that will make this difficult to resolve through purely technical means.

tl:dr; Contacting someone in the abuse department at Yahoo/AOL is probably the best bet to figure this out.

! In T232417#5567208, @aezell wrote:
tl:dr; Contacting someone in the abuse department at Yahoo/AOL is probably the best bet to figure this out.

Yes indeed this looks to be beyond the control of the wmf mail system. I've submitted a yahoo postmaster contact request outlining the issue, and asking for next steps. Hopefully they will help shed light on the cause and next steps.

An incident 01575238 has been created for you. A specialist will email you shortly at postmaster@wikimedia.org

Thanks @Lea_Lacroix_WMDE - I didn't look thoroughly enough at the set of people being affected to recognize this pattern and wasn't aware of this issue at other lists. I did look a bit deeper now, and it looks like you may be right that this is at least mostly accounts that never actually sent an email. Most of these accounts have also similar naming patterns (some long name with a three digit number). I also confirmed that there are a few yahoo.com subscribers left, so not /everyone/ gets unsubscribed.

@Platonides can you confirm that this hypothesis also holds with your situation?

That doesn't mean there are no issues with yahoo though (see @Paladox ' comment above) but it's not of the type that I feared but rather of the same that we have been experiencing for some years now...

@Effeietsanders Yep, that was a similar pattern for me: a name that seems "normal" followed by a 5 digits number. I got several thousands suddenly subscribing to the Wikidata ML in May-June this year and all of them automatically removed by Mailman a few weeks later. I don't think I reported in on Phab, my bad :)

The attack in itself was rather harmless because none of these addresses tried to send some spam - probably because they don't exist. So it was only a hassle for the mailing-list admins receiving hundreds of emails.

There's also a thread on Wikimedia-l from May, 8th 2018 about the same spam attack happening on other Wikimedia MLs, so the issue is not new.

If you find any tips to prevent this kind of attacks in the future (appart from blocking completely subscriptions to our open mailing-lists, which would be unfortunate but why not as a desperate mesure), I'm heavily interested :)

FYI, a new "AOL spam attack" started yesterday on the wikidata mailing-list. I tried to counter it by changing the subscribe_policy parameter to "confirm", but that didn't have an impact, hundred of new fake addresses still subscribing.

Should I create a separate ticket for this, is there chance someone who could help?

I received about 1000 mails in two days, glad that I set up a filter...