Page MenuHomePhabricator

Get mail relay out of Yahoo! blacklist: apply to Yahoo for whitelisting bulk mail
Open, NormalPublic

Description

X-YahooFilteredBulk: 208.80.152.133

(from https://en.wikipedia.org/?oldid=579580929#WMF_notification_email_marked_as_spam_by_Yahoo )

According to several comments, this means that our mail is currently blacklisted by Yahoo!, so it's sent in the "Bulk" folder for users who have it enabled (if not to spam?). WMF needs to send a "Bulk sender application for whitelisting" http://help.yahoo.com/kb/index?page=content&id=SLN9042 but first their guidelines have to be respected: http://help.yahoo.com/kb/index?y=PROD_MAIL_ML&page=content&id=SLN3435

There are several things in there that we don't do; let's add each of them to blockers when we're sure there's something to do.

  • "Use email authentication such as DKIM": low-hanging fruit, let's do it as soon as possible (bug 56413).
  • "Be CAN-SPAM compliant": it's USA law, I assume we're respecting it. :) It seems very lenient, from a quick skimming we're ok. http://business.ftc.gov/documents/bus61-can-spam-act-compliance-guide-business
  • "Consistently manage your lists": no idea, but MediaWiki surely doesn't remove/disable addresses which bounce: I remember Brion mentioning this in some 2005 bug comment or something like that, I don't know if there's an open bug but I'd say it's something worth doing.
  • "Segregate your mail streams by IP address and/or domain": in theory we do, I can't judge if well enough.
  • "Using your domain consistently": no idea, is it consistent to send from wiki@wikimedia.org but then use @wikipedia.org, @mediawiki.org and so on to From headers etc.?
  • "Publish reverse DNS (PTR) records for your sending IPs": looks ok.

$ dig -x 208.80.152.133
;; ANSWER SECTION:
133.152.80.208.in-addr.arpa. 3552 IN PTR wiki-mail.wikimedia.org.

  • "Secure your mail servers": I hope so.
  • "Use common-sense settings": it would be nice to track this (Ori, do you know something?), but Echo sends only few thousands notifications per day on en.wiki (https://toolserver.org/~dartar/en/echo/ ) and standard enotif (talk and watchlist) on all wikis is probably only one or two order of magnitudes bigger, it's still a ridiculously low amount of emails, we're not DoS'ing Yahoo!. It's possible that we have peaks, for instance an edit to a heavily watchlisted pages could cause a peak of thousands enotifs in a single second, but I doubt it's relevant. When we enabled enotifwatchlist on all wikis, the increased network activity of mchenry wasn't even visible at all in ganglia.

Version: unspecified
Severity: major
See Also:
T60261: From: address defaults to mediawiki.org domain on all non-Wikipedia wikis
T59890: [stats] Monitor amount of messages delivered by mail relays (for enotif and everything)
T61731: mailman emails taking long time for delivery, getting stuck in sodium
T66795: Email server's DMARC config prevents users from sending emails via Special:EmailUser
T66818: Mitigate strict DMARC policy on the mailing lists

Details

Reference
bz56414

Related Objects

StatusAssignedTask
OpenNone
ResolvedNone
InvalidNone
Declinedchasemp
OpenNone
Resolved01tonythomas
Resolved01tonythomas
Resolved01tonythomas
Resolved01tonythomas
ResolvedNone
Resolved01tonythomas
InvalidNone
Resolvedcsteipp
Resolved01tonythomas
Resolvedaaron
Resolved01tonythomas
ResolvedNone
Declined01tonythomas
Resolved01tonythomas
OpenNone
ResolvedNone
OpenNone
Resolvedherron

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 2:25 AM
bzimport set Reference to bz56414.
bzimport added a subscriber: Unknown Object (MLST).
Nemo_bis created this task.Oct 31 2013, 8:19 AM
scfc added a comment.Dec 28 2013, 2:33 PM

What's the status of the RT?

(In reply to comment #2)

What's the status of the RT?

No activity, and no owner set.

I have upped the importance, because these delayed emails are causing major issues, and mailman is stalling as a result. Just 2 days ago, when mailman kicked back up, I received a flood of almost 50 emails from various lists. Please get this fixed ASAP. I don't want to listen to music and have my email client make incoming mail noises 47 times during one song.

What's the status of the RT now?

(In reply to comment #4)

I don't want to listen to music and have my email client make incoming mail
noises 47 times during one song.

Whether it's IRC pings or SMS dings, a happier life is quieter things. :-)

Need to find out which exact guidelines are violated.

(Re comment 4: I rather recommend to fix your email client.)

Uhm, I hadn't realized this is blocking bug 59731. I'm sorry; resetting to high priority again.

My email client flooded again just now and a few hours ago.

I'm seeing this issue getting worse, and there is nothing more annoying than leaving for 15 minutes and coming back to a flooded inbox, containing newly received emails sent from as far back as 7 days ago. This is also blocking another bug where sodium tends to fail to send messages as a result of this. I have therefore upped this bug's severity to blocker.

scfc added a comment.Feb 9 2014, 4:45 AM

Please read [[mw:Bugzilla/Fields#Severity]] before changes. Apparently mail is not lost, but just delayed.

(In reply to comment #13)

Please read [[mw:Bugzilla/Fields#Severity]] before changes. Apparently mail
is
not lost, but just delayed.

Well in that case restoring to what it was.

Any updates? I'm getting really close to unsubscribing.

(In reply to comment #15)

Any updates? I'm getting really close to unsubscribing.

Did you write to Yahoo!'s customer service too, out of curiosity? If you can do something without WMF, just do it!

I tries clicking contact us and I just went in circles.

You should see my ISP's! It's easy: https://io.help.yahoo.com/contact/index?page=contact&locale=en_US&y=PROD_MAIL_ML -> Sending and receiving messages (last) -> Unable to receive messages (third last) -> Email

Note however that nowadays it's easier to receive support from email providers on Twitter, I see they're very active there: https://twitter.com/YahooCare

You can also do some lobbying on their uservoice board; the only relevant topic I found is https://yahoo.uservoice.com/forums/210695-yahoo-mail/suggestions/5361666-final-solution-to-prevent-spam which proposes to whitelist wikipedia.org among other things.

And of course I assume you went through https://help.yahoo.com/kb/mail/SLN3517.html

On a related note, I just ran into German article about the problems of people with a Yahoo mail account on http://www.heise.de/newsticker/meldung/DMARC-Policy-Yahoo-killt-Mailinglisten-Mitgliedschaften-2168857.html (and Google Translate to English pretty much works).
For the records, Mailman 2.1.16 (which will be in Ubuntu Trusty) allows setting "FROM_IS_LIST=yes" to "replace the sender with the list address to conform with policies like ADSP and DMARC".

(In reply to Andre Klapper from comment #20)

On a related note, I just ran into German article about the problems of
people with a Yahoo mail account

That's a new thing though, technical explanation at http://www.ietf.org/mail-archive/web/ietf/current/msg87153.html

Seb35 added a comment.Apr 30 2014, 3:11 PM

I currently study the Yahoo and AOL’s recent change to DMARC, and the problem is pretty different than the one described here, basically because this one doesn’t relay emails (as do a mailing list) but sends original mails.

Possibly it could be implemented DMARC(+DKIM+SPF) on wikimedia.org to improve delivrability for this bug, *BUT* this could lead to heavy consequences on other @wikimedia.org emails (like non-delivery) so it must be carefully thought before action.

(In reply to Nemo from comment #21)

(In reply to Andre Klapper from comment #20)

On a related note, I just ran into German article about the problems of
people with a Yahoo mail account

That's a new thing though, technical explanation at
http://www.ietf.org/mail-archive/web/ietf/current/msg87153.html

That was split to bug 64795.

Bug 56413 was fixed, so I'm told this is currently waiting for bug 46640 and mailman 2.1.17+ (for MediaWiki mailer's and mailman's IPs respectively).

  • Bug 70930 has been marked as a duplicate of this bug. ***
Nemo_bis set Security to None.
Dzahn removed a subscriber: Dzahn.Dec 22 2014, 2:04 PM
chasemp lowered the priority of this task from High to Normal.Jan 6 2015, 11:29 PM
chasemp added a subscriber: chasemp.

This is over a year old with no assignee. I am setting to normal priority. Please reassess when resources are available.

Qgil removed a subscriber: Qgil.Jan 13 2016, 3:01 AM

As far as I can see everything is ready here and T66795#2867163 should reduce bounces to/from Yahoo (although it was arguably their fault, not ours); can ops please apply for said whitelisting? Looks like it's now called "reputation review" and the form is linked from https://help.yahoo.com/kb/SLN24439.html#cont3

Scoopfinder added a subscriber: Scoopfinder.

Not sure if related but emails from wikimedia our taking super long for me compared to using outlook.

It seems i managed to get yahoo to unblock wikimedia though unsure if i did. Anyways my mail is back to normal from wikimedia :)

revi added a subscriber: revi.Mar 12 2018, 12:57 AM