Page MenuHomePhabricator

Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines
Closed, ResolvedPublic

Description

As a followup to T355449: Ensure wikimedia.org adheres to Google's sender guidelines, where @thcipriani wrote:

I realize you mentioned Phab and Gerrit are out of scope, but for the purposes of gathering data: looking at exim logs, Phab sent 10,740 emails to gmail addresses yesterday and 7195 today (since we did the phab upgrade on Saturday, I only have two days of historical data on disk there). Gerrit has hovered between 2k–3k emails to gmail addresses per day.

From sender guidelines:

If you send more than 5,000 message per day, your marketing and subscribed messages must support one-click unsubscribe.

To set up one-click unsubscribe, include both of these headers in outgoing messages:

Phab/Phorge mails sent have a preference management link in the body, but no List-Unsubscribe, let alone List-Unsubscribe-Post. A quick search didn't find anything about this upstream.

Requirements for all senders

  • Set up SPF or DKIM email authentication for your domain.
  • Ensure that sending domains or IPs have valid forward and reverse DNS records, also referred to as PTR records.
  • Use a TLS connection for transmitting email.
  • Keep spam rates reported in Postmaster Tools below 0.10% and avoid ever reaching a spam rate of 0.30% or higher.
  • Format messages according to the Internet Message Format standard (RFC 5322).
  • Don’t impersonate Gmail From: headers. Gmail will begin using a DMARC quarantine enforcement policy, and impersonating Gmail From: headers might impact your email delivery.
  • If you regularly forward email, including using mailing lists or inbound gateways, add ARC headers to outgoing email.

Requirements for high-volume senders

I don’t have data on this atm but I would not be surprised if we’re over the 5k emails per day threshold.

  • Set up DMARC email authentication for your sending domain. Your DMARC enforcement policy can be set to none.
  • For direct mail, the domain in the sender’s From: header must be aligned with either the SPF domain or the DKIM domain. This is required to pass DMARC alignment.
  • Marketing messages and subscribed messages must support one-click unsubscribe, and include a clearly visible unsubscribe link in the message body.

Event Timeline

Without having dug into the code at all yet, I expect that a List-Unsubscribe just pointing to the existing https://phabricator.wikimedia.org/settings/panel/emailpreferences/ would be simple, while an actual one-click unsubscribe will be a great deal more involved. Ideally this would be handled for upstream in general. I'll file something there.

upstream ticket comment pointed out we can run ./mail/volume from /srv/phab/phabricator/bin to check how many mails are being sent.

quote " It just collects all mails created in the last N days and counts them by user.".

currently running that

edit: but it failed like this:

@phab1004:/srv/phab/phabricator/bin# ./mail volume

Killed

@jhathaway and I took a look through, and I've updated the checklist above. The tl;dr is that we're looking ok from the phabricator side. We should probably remove the existing spf records that point to ip6:2620:0:861:102:10:64:16:101 ip6:2620:0:860:103:10:192:32:54, since these are essentially redundant but also don't resolve publicly so might run foul of the requirement to have records forward-and-reverse resolvable.

Is it known/ok that Phab is sending mail from phabricator.wikimedia.org but the DKIM signature is for wikimedia.org?

There shouldn't be anything directly sending from phabricator.wm.o, it should route through the mx* hosts

Is it known/ok that Phab is sending mail from phabricator.wikimedia.org but the DKIM signature is for wikimedia.org?

I don't see a dkim signature in the emails I have from phabricator, can you attach an example?

Interesting, it seems like Phabricator only signs mail sent to non-wikimedia.org addresses. The From: header is no-reply@phabricator.wikimedia.org and DKIM signature uses d=wikimedia.org. Here is an example: {F41732140}

ah, that seems like a bug? However, dkim signature should be okay, since our dmarc record is set to relaxed. Which allows subdomain matching, i.e. dkim: wikimedia.org aligns with from: no-reply@phabricator.wikimedia.org.

Change 994795 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/dns@master] phabricator: verify domain for Google's postmaster tools

https://gerrit.wikimedia.org/r/994795

Change 994795 merged by JHathaway:

[operations/dns@master] phabricator: verify domain for Google's postmaster tools

https://gerrit.wikimedia.org/r/994795

We've ticked all the boxes here for the most part. The two outstanding items are monitoring spam rates and one-click unsubscribe.

The spam rate isn't visible in postmaster tools, which we're attributing to a low spam rate. The one-click unsubscribe is currently with upstream but probably won't gain any traction.

Either way, there's nothing immediately left for us to take care of here.