wiki-mail DKIM failing
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krenair
	Apr 17 2019, 8:42 PM

Description

templates/wikimedia.org

wiki-mail._domainkey    1H  IN TXT  "v=DKIM1; k=rsa; g=wiki*; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDOZG4udzEAJiq1D0TG+5BYil0vuh8/iuBmsrmAZQcWYaMcBIlthxg0a/5xktMpE4A6w135mgl/q61dbDnT6d3Y3veWMOy0xI0q/wS9DCmRqU2VIb7L/Asjp9evwzEJywODigjOohkk4+/aIbppU1/GLGCoc6b9h7m0iandPkKTXwIDAQAB"

But, on a mail I got this morning, gmail reports DKIM failure:

Full headers:

P8416 Email headers for DKIM failure task

1	Delivered-To: krenair@gmail.com
2	Received: by 2002:a9f:2383:0:0:0:0:0 with SMTP id 3csp5077617uao;
3	Wed, 17 Apr 2019 03:59:29 -0700 (PDT)
4	X-Google-Smtp-Source: APXvYqyOpwp/Y7yy6G0cOUvlEpabN0QDTVoEvXJzDteBorJLG12QmcSZspzK7GByjpnFqZS2qBQ6
5	X-Received: by 2002:ac8:29f6:: with SMTP id 51mr73525854qtt.65.1555498769014;
6	Wed, 17 Apr 2019 03:59:29 -0700 (PDT)
7	ARC-Seal: i=1; a=rsa-sha256; t=1555498769; cv=none;
8	d=google.com; s=arc-20160816;
9	b=1D46fYC3GqFdRfcPW12fkH2BkZCDS4lwJo89shs9ymef5hgNDdCyrWWE2Kmto21esd
10	bUeVW6eHhaBZhkBSv0mPbY9C225A3Pqywpztr5LiSRqTB+ix0kw34aXWRG+0hga9kxPr
11	NPdGt6BvVfudvT040eugfJhSgZ6TDmCDz7TBd8FM3bYYBT3UXOeyYyq2evjFo5Lfh4Bq
12	herBKO5mNNE4yceBGiEqW2DbtO/2EagrC7jOajHiWuyZI2ydl4CgmKmyuJmgGZOJwMHM
13	Jd39HljQXqu8awXlvN714Cynf5svMWUQovKKxnGDg/c4D+O5r2N43/wx+xGgIXcpSpRy
14	xfcA==
15	ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
16	h=content-transfer-encoding:mime-version:list-unsubscribe:message-id
17	:date:reply-to:from:list-help:subject:to:dkim-signature;
18	bh=nepsm00dXvBIviICxocvbDXaPvgaEWsD9GuK806JLdE=;
19	b=pGwhytGOYvxzQvxxsMRO5VhXPOCLcF9MlHGZsVNXvGjgbTGGseVyubNMZ3V2TRkLX1
20	j0yvt66ziQHv1ce+cYVTMysAzslAhIUvc6X20jQwaj42AQ9nNwjtGYpEAvBKdOsn/i33
21	r/34AyokM5ZhZ/Fbn5qBnjNV593KPrgfnSUlk1feWwOeAQAktaQMfjEdsfzigA46H8DZ
22	Zfd802DvoyN5xt5e2qq12DpC2jy/O9AKskgG4mPFyvp8/Paa3x6TCF+ZpCG+YgFxBXiI
23	MavRbt8cDwEdtDfPeRCoiwApv81uzWirdLcvQr5wwEx9lwnklRq9ecqU+IrmwnqC0gq7
24	y+nA==
25	ARC-Authentication-Results: i=1; mx.google.com;
26	dkim=neutral (no key) header.i=@wikimedia.org header.s=wiki-mail header.b=jTAIvcJs;
27	spf=pass (google.com: domain of wiki-metawiki-oz58-pq3qj4-megamfdrlhuo670q@wikimedia.org designates 208.80.154.91 as permitted sender) smtp.mailfrom=wiki-metawiki-oz58-pq3qj4-MegamfDRLHuO670q@wikimedia.org;
28	dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org
29	Return-Path: <wiki-metawiki-oz58-pq3qj4-MegamfDRLHuO670q@wikimedia.org>
30	Received: from wiki-mail-eqiad.wikimedia.org (wiki-mail-eqiad.wikimedia.org. [208.80.154.91])
31	by mx.google.com with ESMTPS id b125si3319279qke.208.2019.04.17.03.59.28
32	for <krenair@gmail.com>
33	(version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
34	Wed, 17 Apr 2019 03:59:29 -0700 (PDT)
35	Received-SPF: pass (google.com: domain of wiki-metawiki-oz58-pq3qj4-megamfdrlhuo670q@wikimedia.org designates 208.80.154.91 as permitted sender) client-ip=208.80.154.91;
36	Authentication-Results: mx.google.com;
37	dkim=neutral (no key) header.i=@wikimedia.org header.s=wiki-mail header.b=jTAIvcJs;
38	spf=pass (google.com: domain of wiki-metawiki-oz58-pq3qj4-megamfdrlhuo670q@wikimedia.org designates 208.80.154.91 as permitted sender) smtp.mailfrom=wiki-metawiki-oz58-pq3qj4-MegamfDRLHuO670q@wikimedia.org;
39	dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org
40	DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=wikimedia.org; s=wiki-mail; h=Content-transfer-encoding:Content-type: MIME-Version:List-Unsubscribe:Message-ID:Date:Reply-To:From:List-Help:Subject :To:Sender:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Id:List-Subscribe:List-Post:List-Owner:List-Archive; bh=nepsm00dXvBIviICxocvbDXaPvgaEWsD9GuK806JLdE=; b=jTAIvcJsNjbkIY6Gq8+LSKFavk tG3Gi5ALy4aM9T5V1Mfmsz2gX1enxTIOYembewDHq0i2h+oWfC+bCR91gR+o2NUgjetGo3J5sPIiX s54sVQo3HFHQ4xH7dvE64maVBdatHeHVyVxVPgCMBk43uB+kGfd3sW8cI5yCN9LgowsY=;
41	Received: from [2620:0:861:103:1a66:daff:fe99:52f8] (port=59018 helo=mw1337.eqiad.wmnet) by mx1001.wikimedia.org with esmtp (Exim 4.89) (envelope-from <wiki-metawiki-oz58-pq3qj4-MegamfDRLHuO670q@wikimedia.org>) id 1hGiHo-0004vw-RW for krenair@gmail.com; Wed, 17 Apr 2019 10:59:28 +0000
42	Received: from www-data by mw1337.eqiad.wmnet with local (Exim 4.89) (envelope-from <wiki-metawiki-oz58-pq3qj4-MegamfDRLHuO670q@wikimedia.org>) id 1hGiHo-0000Kw-OB for krenair@gmail.com; Wed, 17 Apr 2019 10:59:28 +0000
43	To: Krenair <krenair@gmail.com>
44	Subject: Meta page Talk:OTRS has been changed by Bluerasberry
45	List-Help: https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Watchlist
46	From: Meta <wiki@wikimedia.org>
47	Reply-To: wiki@wikimedia.org
48	Date: Wed, 17 Apr 2019 10:59:28 +0000
49	Message-ID: <metawiki.5cb707106aee45.76621288@meta.wikimedia.org>
50	X-Mailer: MediaWiki mailer
51	List-Unsubscribe: <https://meta.wikimedia.org/wiki/Special:Preferences>
52	MIME-Version: 1.0
53	Content-type: text/plain; charset=UTF-8
54	Content-transfer-encoding: 8bit

@GedHaywood reports they got DKIM failures from wiki-mail due to a granularity mismatch

Details

	Subject	Repo	Branch	Lines +/-
	remove granularity key from wiki-mail DKIM	operations/dns	master	+1 -1

Customize query in gerrit

Related Objects

Mentioned Here: P8416 Email headers for DKIM failure task

Event Timeline

Krenair created this task.Apr 17 2019, 8:42 PM

Restricted Application added projects: SRE, Traffic. · View Herald TranscriptApr 17 2019, 8:42 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Indeed I'm able to produce a DKIM issue as well with wiki-mail. Here's an example (seen in headers of message triggered by account preferences change):

dkim=invalid (public key: granularity mismatch, 1024-bit rsa key sha256)
      header.d=wikimedia.org header.i=@wikimedia.org header.b=kAFt/wB3
      header.a=rsa-sha256 header.s=wiki-mail x-bits=1024;

We're currently specifying g=wiki* within the wiki-mail dkim record

wiki-mail._domainkey.wikimedia.org descriptive text "v=DKIM1; k=rsa; g=wiki*; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDOZG4udzEAJiq1D0TG+5BYil0vuh8/iuBmsrmAZQcWYaMcBIlthxg0a/5xktMpE4A6w135mgl/q61dbDnT6d3Y3veWMOy0xI0q/wS9DCmRqU2VIb7L/Asjp9evwzEJywODigjOohkk4+/aIbppU1/GLGCoc6b9h7m0iandPkKTXwIDAQAB"

However in the dkim output about I see header.i=@wikimedia.org (empty user-part) and a mismatch between g= and i= would explain what we're seeing.

For quick reference:

g= Granularity of the key (plain-text; OPTIONAL, default is "*"). This value MUST match the Local-part of the "i=" tag of the DKIM-Signature header field (or its default value of the empty string if "i=" is not specified)

i= Identity of the user or agent (e.g., a mailing list manager) on behalf of which this message is signed (dkim-quoted-printable; OPTIONAL, default is an empty Local-part followed by an "@" followed by the domain from the "d=" tag)

This looks to be the likely cause, and presumably statuses of "invalid" and "neutral (no key)" are a result of the same condition as interpreted by differing mail software.

The question that comes to my mind is what was the history of this granularity setting, and was it previously working as expected? I found a commit in the operations/dns repo dating back to Aug 2013, but it doesn't explain very much.

commit f66697ba390fa56eff9773e4993c1b3a33594326
Author: root <root@wikimedia.org>
Date:   Tue Aug 20 04:03:02 2013 +0000

    g=wiki (local part) for wiki-mail DKIM

For a near-term fix I'd be inclined to remove the granularity key from the wiki-mail record.

Change 504948 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/dns@master] remove granularity key from wiki-mail DKIM

https://gerrit.wikimedia.org/r/504948

gerritbot added a project: Patch-For-Review.Apr 18 2019, 7:05 PM

colewhite claimed this task.Apr 18 2019, 7:07 PM

colewhite triaged this task as High priority.

How did it work until now?

Also, unrelatedly, we probably should use something stronger than a 1024-bit RSA key.

In T221290#5123622, @faidon wrote:

How did it work until now?

I wonder the same thing. Looking through old personal emails I have a message from wiki@wikimedia.org dated Sep 18 2018 which has the same issue:

Date: Tue, 18 Sep 2018 17:52:18 +0000
    dkim=invalid (public key: granularity mismatch, 1024-bit rsa key sha256)
      header.d=wikimedia.org header.i=@wikimedia.org header.b=yW/crbag
      header.a=rsa-sha256 header.s=wiki-mail x-bits=1024;

Maybe someone has old archived messages from wiki@wikimedia.org and could chime in with the date and dkim headers.

It's been a while but if I recall correctly, the intention was to not allow (= not create a valid signature) emails that had e.g. From: person@wikipedia.org (where person = jimmy for instance), when those emails originated from the MW appserver fleet.

The idea was to limit wiki-mail's signature's validity to only wiki@wikimedia.org (later wiki + VERP, hence the wildcard), which is the only address that should originate from there.

The original spec (e.g. here: http://dkim.org/specs/draft-allman-dkim-base-01.html) said this instead:

granularity of the key (plain-text; OPTIONAL, default is "*"). This value MUST match the local part of the signing address, with a "*" character acting as a wildcard. The intent of this tag is to constrain which signing address can legitimately use this selector. An email with a signing address that does not match the value of this tag constitutes a failed verification. Wildcarding allows matching for addresses such as "user+*". An empty "g=" value never matches any addresses.

I'm not sure what i= is, or how the spec evolved since then. If we could keep the behavior listed above that'd be ideal, but if it's much work I think we can soften our requirements -- it was a bit strict in nature anyway.

In T221290#5123696, @herron wrote:
In T221290#5123622, @faidon wrote:

How did it work until now?

I wonder the same thing. Looking through old personal emails I have a message from wiki@wikimedia.org dated Sep 18 2018 which has the same issue:
Date: Tue, 18 Sep 2018 17:52:18 +0000
    dkim=invalid (public key: granularity mismatch, 1024-bit rsa key sha256)
      header.d=wikimedia.org header.i=@wikimedia.org header.b=yW/crbag
      header.a=rsa-sha256 header.s=wiki-mail x-bits=1024;
Maybe someone has old archived messages from wiki@wikimedia.org and could chime in with the date and dkim headers.

Done some digging, gmail doesn't show a DKIM line (above real mail headers gmail provides some metadata including DKIM, SPF, DMARC etc. status of the mail - see screenshot in task description) for this one:

Received: from wiki-mail-eqiad.wikimedia.org (wiki-mail-eqiad.wikimedia.org. [208.80.154.91])
        by mx.google.com with ESMTP id n5si17140591qha.111.2015.06.02.14.54.26
        for <krenair@gmail.com>;
        Tue, 02 Jun 2015 14:54:26 -0700 (PDT)
Received-SPF: pass (google.com: domain of wiki-metawiki-oz58-npc7ip-FRNZztnZ/wesOhN3@wikimedia.org designates 208.80.154.91 as permitted sender) client-ip=208.80.154.91;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of wiki-metawiki-oz58-npc7ip-FRNZztnZ/wesOhN3@wikimedia.org designates 208.80.154.91 as permitted sender) smtp.mail=wiki-metawiki-oz58-npc7ip-FRNZztnZ/wesOhN3@wikimedia.org;
       dkim=neutral (no key) header.i=@;
       dmarc=pass (p=NONE dis=NONE) header.from=wikimedia.org
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=wikimedia.org; s=wiki-mail; h=Content-transfer-encoding:Content-type:MIME-Version:Message-ID:Date:Reply-To:From:Subject:To; bh=Ftv+zV0SPtGZlEoSvuwR6iSCYI3HMDgqJHaj241cZhY=; b=v52Wup/SCeAAx0wR/U/eWI03JQ7Ozbo/IH5KGLDepmQ3XSc8PPy/BRp8cJ8VvOZRB5Lvs32h+j9oCzIa+A16wlgihiBk7nlHkkjL6SYznCf/rOB8DRXVJ9/FjHRPmFLlV7x9WW9DZBpD8ZlhnRZ5iaj0ir1b7OHJadt3EsD/+cc=;
Received: from [2620:0:861:101:7a2b:cbff:fe08:7b5a] (port=38871 helo=mw1015.eqiad.wmnet) by polonium.wikimedia.org with esmtp (Exim 4.82) (envelope-from <wiki-metawiki-oz58-npc7ip-FRNZztnZ/wesOhN3@wikimedia.org>) id 1Yzu8g-0001QZ-2u for krenair@gmail.com; Tue, 02 Jun 2015 21:54:26 +0000
Received: from www-data by mw1015.eqiad.wmnet with local (Exim 4.82) (envelope-from <wiki-metawiki-oz58-npc7ip-FRNZztnZ/wesOhN3@wikimedia.org>) id 1Yzu8g-0005aw-07 for krenair@gmail.com; Tue, 02 Jun 2015 21:54:26 +0000

First instance of DKIM: 'FAIL' with domain wikimedia.org in my gmail from wiki@wikimedia.org:

Received: from wiki-mail-eqiad.wikimedia.org (wiki-mail-eqiad.wikimedia.org. [208.80.154.91])
        by mx.google.com with ESMTP id r10si1129842qkh.86.2015.06.03.09.43.04
        for <krenair@gmail.com>;
        Wed, 03 Jun 2015 09:43:05 -0700 (PDT)
Received-SPF: pass (google.com: domain of wiki-metawiki-oz58-npdnrs-hem5rurNyw4QvUfX@wikimedia.org designates 208.80.154.91 as permitted sender) client-ip=208.80.154.91;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of wiki-metawiki-oz58-npdnrs-hem5rurNyw4QvUfX@wikimedia.org designates 208.80.154.91 as permitted sender) smtp.mail=wiki-metawiki-oz58-npdnrs-hem5rurNyw4QvUfX@wikimedia.org;
       dkim=neutral (no key) header.i=@wikimedia.org;
       dmarc=pass (p=NONE dis=NONE) header.from=wikimedia.org
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=wikimedia.org; s=wiki-mail; h=Content-transfer-encoding:Content-type:MIME-Version:Message-ID:Date:Reply-To:From:Subject:To; bh=ZppDHyqmC1HfnpbQid3cnVAIlKdD8JbxpRZ/zPiqXg0=; b=eSKTMBH5pyPD8o3KPMFU5pey+rHdaFm6I1w/5siszmk6OpsMe9xnn5F2ctkLbvFme21rRgcyI8lyXYECdddUExWsBXl6IbbaH2f/NQMzKupK+20MNuwEtCuUtv2e4ffze8FOZn1Xnx+2M6WbTDNh90B3DD/fQvErWHoNtvViAEA=;
Received: from [2620:0:861:101:862b:2bff:fe78:28c8] (port=46580 helo=mw1003.eqiad.wmnet) by polonium.wikimedia.org with esmtp (Exim 4.82) (envelope-from <wiki-metawiki-oz58-npdnrs-hem5rurNyw4QvUfX@wikimedia.org>) id 1Z0Bku-000A6i-OP for krenair@gmail.com; Wed, 03 Jun 2015 16:43:04 +0000
Received: from www-data by mw1003.eqiad.wmnet with local (Exim 4.82) (envelope-from <wiki-metawiki-oz58-npdnrs-hem5rurNyw4QvUfX@wikimedia.org>) id 1Z0Bku-0007jd-LH for krenair@gmail.com; Wed, 03 Jun 2015 16:43:04 +0000

Actual mail headers show header.i in the dkim line changing... Not sure what implications that has. Might've been some change on gmail's end.
Does this explain anything? If not, I could keep digging if I knew more about what to look for.

That's interesting, based on the headers in T221290#5123805 it looks like this issue goes back as far as 2015.

In T221290#5123730, @faidon wrote:

It's been a while but if I recall correctly, the intention was to not allow (= not create a valid signature) emails that had e.g. From: person@wikipedia.org (where person = jimmy for instance), when those emails originated from the MW appserver fleet.

The idea was to limit wiki-mail's signature's validity to only wiki@wikimedia.org (later wiki + VERP, hence the wildcard), which is the only address that should originate from there.

I'm not sure what i= is, or how the spec evolved since then. If we could keep the behavior listed above that'd be ideal, but if it's much work I think we can soften our requirements -- it was a bit strict in nature anyway.

Ok, in that case adjusting exim dkim_identity looks promising

dkim_identity	Use: smtp	Type: string†	Default: unset
If set after expansion, the value is used to set an "i=" tag in the signing header. The DKIM standards restrict the permissible syntax of this optional tag to a mail address, with possibly-empty local part, an @, and a domain identical to or subdomain of the "d=" tag value. Note that Exim does not check the value.

i= Identity of the user or agent (e.g., a mailing list manager) on behalf of which this message is signed (dkim-quoted-printable; OPTIONAL, default is an empty Local-part followed by an "@" followed by the domain from the "d=" tag)

Something to the effect of dkim_identity = ${sender_address_local_part}@${dkim_domain} seems on paper it would do the right thing, in this case set i=wiki@wikimedia.org

Would need some testing, but seems low risk considering a bad value would cause granularity mismatches, and we already are experiencing these.

Krenair updated the task description. (Show Details)Apr 19 2019, 3:53 PM

Krenair added a subscriber: GedHaywood.

• ema moved this task from Backlog to Domain name registration on the Traffic board.Apr 29 2019, 3:53 PM

Change 504948 merged by Herron:
[operations/dns@master] remove granularity key from wiki-mail DKIM

https://gerrit.wikimedia.org/r/504948

Looking better after merging the above. From a password reminder mail:

Date: Wed, 01 May 2019 17:29:44 +0000
    dkim=pass (1024-bit rsa key sha256) header.d=wikimedia.org
      header.i=@wikimedia.org header.b=WvwucqV9 header.a=rsa-sha256
      header.s=wiki-mail x-bits=1024;

Looks better to me too.

herron closed this task as Resolved.May 8 2019, 1:57 PM

wiki-mail DKIM failingClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

wiki-mail DKIM failing
Closed, ResolvedPublic
Actions