Can someone who has some of the emails that were marked as spam please pastebin the headers? We're working with a contact at Google now to diagnose. It is probably the SPF issue, but they haven't suggested it yet (they probably haven't gotten that far).

http://pastebin.com/BTkUj0Rv

(just to be clear: my fault, the Google contact did indeed suggest fixing SPF and DKIM, but I didn't read that far in the email)

What I just sent to the wikitech-l list:

We're talking with a support person from Google, they've made a couple
of suggestions that I've forwarded to Ops (specifically SPF and DKIM).

They also asked for the headers of some of the messages, which you saw I
related to the bug. I sent them the one Alex shared.

I'll let the list know when I have a more solid answer (ie: "we're all
fixed" or "email hates us").

So I think the SPF record to add is the following, however I do not have that much experience in this and I'm not 100%.

Its basically a copy of the wikimedia.org domain spf, except without google info since we relay our own list mail (is my understanding).

I'd like someone who knows to comment (like Mark maybe?)

lists.wikimedia.org. 600 IN TXT "v=spf1 ip4:91.198.174.0/24 ip4:208.80.152.0/22 ip6:2620:0:860::/46 ?all"

Older ticket links for wikimedia.org spf settting:

https://bugzilla.wikimedia.org/show_bug.cgi?id=1609

So going to add Hashar to this to check out what I put, and if its right I'll roll it out.

Hashar, please review and assign back to me with comments, thank you!

Hashar's on vacation until August 25.

Since Antoine is MIA until the end of the month, unassigning from him. Rob: your choice on who's the next assignee ;)

RT #362 seems to have the information. Just copy the TXT record of wikimedia.org to lists.wikimedia.org (with low TTL) that should be fine.

In the previous bug, I think I barely babysitted the bug, not much conducted any action :)

Hashar points out the changes needed are in a linked RT ticket, and I have pasted it.

Jeff has most recently updated the SPF for wikimedia.org, so I'll be chatting with him later today.

Just updating ticket to reflect its not dead, and I am on it ;]

From our google support thread:
"In order to have this issue resolved, the domain lists.wikimedia.org should
update the SPF record to include that IP address in it. It should include :
v=spf1 ip6:2620:0:861:1::2 include:_spf.google.com ~all"

Yea, I am leaving in the google stuff and basically making lists.wikimedia.org records for txt/spf identical to wikimedia.org.

Ok, I just rolled out the change for this, and lists.wikimedia.org now has identical SPF settings to our wikimedia.org top level domain. Which makes sense since all the same mail handlers are involved ;]

Resolving this ticket.

Correction: for the record, i had it wrong.

I stripped out all the IP info and just left in the "v=spf1 a mx ~all" part in its place. this says that the spf sender is the defined mx servers for the domain.

Are you sure all outgoing emails go out from our MX ? Just wondering.

So once these were in place we were finding more issues than before. It seems the spam marking isn't due to this, so its being reverted for now.

paravoid posted two traces to http://tty.gr/s/google-ipv6-spf.txt. Apparently, Google doesn't test the host it received the mail from against the SPF rules, but a host in the Received: chain before that. So this looks like a (major) bug on Google's part. It would be nice if this could be communicated to them so they can fix it.

Created attachment 13068
Example almost-full headers with SPF records on

Just for the records, the SPF check did pass before the new SPF records were removed, see attachment.

It's also true that lists have not passed SPF checks on Google for over a year now (they did till May/summer 2012, then not) and the reported spam filters problems are very recent, so unless Google changed something in how they deal with neutral SPF results the two seem unrelated.

Attached:

@Rob -- does comment 14 mean that an incorrect SPF record was published for some amount of time? Perhaps the issues in comment 16 are due to the incorrect record being cached?

Also: the headers posted by paravoid in comment 17 seem to indicate difficulty with IPv6 addresses. Did our SPF correctly include the IPv6 of our senders?

Created attachment 13073
Example headers after 21:26 paravoid: authdns-update: adding lists AAAA & (neutral) SPF

Today all ML messages I receive have "spf=pass", checked with Thunderbird's headers search; 15/37, sent by some (not all) gmail users, also have

dkim=neutral (bad format) header.i=@gmail.com;
dmarc=fail (p=NONE dis=NONE) d=gmail.com

Attached:

$ dig lists.wikimedia.org TXT

; <<>> DiG 9.8.3-P1 <<>> lists.wikimedia.org TXT
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29211
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;lists.wikimedia.org. IN TXT

;; ANSWER SECTION:
lists.wikimedia.org. 3600 IN TXT "v=spf1 mx ?all"

;; Query time: 69 msec
;; SERVER: 10.0.1.1#53(10.0.1.1)
;; WHEN: Tue Aug 20 10:41:03 2013
;; MSG SIZE rcvd: 64

Fairly sure this is resolved/fixed, not resolved/wontfix.

I don't think the change was puppetized, but just hot-fixed, so the SPF records will be lost with the next change.

(In reply to comment #23)

I don't think the change was puppetized, but just hot-fixed, so the SPF
records
will be lost with the next change.

What's this then? https://git.wikimedia.org/commit/operations%2Fdns.git/6040cf220b9c8fe22f2557915257009f31f29ae4

Related and more recent:
Change-Id: I0d24518dd0f1168b2d50c783cce4f3b1354cdf7c
Change-Id: I42a63a82ff4c411fd2378f1bf1b0383b46b8b728
Change-Id: I7cebaa9a30a8859d4244ba5b815afb5cdeb3d08c

(In reply to comment #24)

I don't think the change was puppetized, but just hot-fixed, so the SPF
records
will be lost with the next change.

What's this then?
<https://git.wikimedia.org/commit/operations%2Fdns.git/
6040cf220b9c8fe22f2557915257009f31f29ae4>

[...]

I'm sorry, I missed this. There were no pointers in #wikimedia-operations when the change was discussed, it wasn't linked here, I didn't (and don't) see any DNS repo on noc.wikimedia.org, and Rob said in comment #16:

So once these were in place we were finding more issues than before. It
seems
the spam marking isn't due to this, so its being reverted for now.

and changed the status from FIXED to WONTFIX. "Don't trust anybody, Dr Jones."