Page MenuHomePhabricator

Outdated TLS config for MXes
Open, MediumPublic

Description

The TLS config of our MXes is outdated by modern standards:

  • Supports TLS 1.0
  • Outdated ciphers
  • No PFS
  • No OCSP stapling
  • (probably others)

We should address that ASAP, although it's probably better to wait for the upgrade to stretch (T175361) to complete first. For example, OCSP stapling was attempted and implemented before, but reverted with 90dbb023366cc761073f1b15edb37ccc33fd49f9 because of implementation bugs with jessie's exim version.

Note that exim4 uses GnuTLS, so both its cipher support and its configuration etc. is a bit different.

Event Timeline

faidon triaged this task as Medium priority.Aug 31 2018, 3:46 PM
faidon created this task.

Change 458061 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] mx: strengthen exim tls_require_ciphers

https://gerrit.wikimedia.org/r/458061

Upgrading mx1001 to stretch first makes sense to me as well. Comparing mx2001 (stretch) to mx1001 (jessie), upgrading to stretch itself addressed RC4 and TLS_FALLBACK_SCSV issues (for starters) without modifying the Exim config.

Above is an updated Exim TLS priority string for review. My thinking was to merge after upgrading to stretch (along with re-enabling OCSP stapling) but if needed we should be able to deploy it sooner.

Regarding deployment, there may be something we can do with server side ciphersuite precedence to try and reduce the number of TLS1.0/TLS1.1 clients as much as possible prior to dropping support.

And of course post-deployment we can monitor for spikes in TLS error rates with https://grafana.wikimedia.org/dashboard/db/mail?refresh=5m&panelId=24&fullscreen&orgId=1

Change 458289 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] mtail: add exim tls ciphersuite metrics

https://gerrit.wikimedia.org/r/458289

Change 458289 merged by Herron:
[operations/puppet@production] mtail: add exim tls ciphersuite metrics

https://gerrit.wikimedia.org/r/458289

With the upgrade to stretch complete here is a snapshot of current mx1001 TLS ciphers, protocols, etc. (output from testssl)

1 Start 2018-09-14 14:59:06 -->> 208.80.154.76:25 (mx1001.wikimedia.org) <<--
2
3 further IP addresses: 2620:0:861:3:208:80:154:76
4 rDNS (208.80.154.76): mx1001.wikimedia.org.
5 Service set: STARTTLS via SMTP
6
7 Testing protocols via sockets
8
9 SSLv2 not offered (OK)
10 SSLv3 not offered (OK)
11 TLS 1 offered
12 TLS 1.1 offered
13 TLS 1.2 offered (OK)
14 SPDY/NPN (SPDY is an HTTP protocol and thus not tested here)
15 HTTP2/ALPN (HTTP/2 is a HTTP protocol and thus not tested here)
16
17
18 Testing ~standard cipher categories
19
20 NULL ciphers (no encryption) not offered (OK)
21 Anonymous NULL Ciphers (no authentication) not offered (OK)
22 Export ciphers (w/o ADH+NULL) not offered (OK)
23 LOW: 64 Bit + DES encryption (w/o export) not offered (OK)
24 Weak 128 Bit ciphers (SEED, IDEA, RC[2,4]) not offered (OK)
25 Triple DES Ciphers (Medium) offered
26 High encryption (AES+Camellia, no AEAD) offered (OK)
27 Strong encryption (AEAD ciphers) offered (OK)
28
29
30 Testing robust (perfect) forward secrecy, (P)FS -- omitting Null Authentication/Encryption, 3DES, RC4
31
32 PFS is offered (OK) ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-CHACHA20-POLY1305 DHE-RSA-CHACHA20-POLY1305 DHE-RSA-AES256-CCM DHE-RSA-AES256-SHA256
33 DHE-RSA-AES256-SHA ECDHE-RSA-CAMELLIA256-SHA384 DHE-RSA-CAMELLIA256-SHA256 DHE-RSA-CAMELLIA256-SHA TLS_DHE_RSA_WITH_CAMELLIA_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_CAMELLIA_256_GCM_SHA384
34 ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-CCM DHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA ECDHE-RSA-CAMELLIA128-SHA256
35 DHE-RSA-CAMELLIA128-SHA256 DHE-RSA-CAMELLIA128-SHA TLS_DHE_RSA_WITH_CAMELLIA_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_CAMELLIA_128_GCM_SHA256
36 Elliptic curves offered: prime192v1 secp224r1 prime256v1 secp384r1 secp521r1
37
38
39 Testing server preferences
40
41 Has server cipher order? nope (NOT ok)
42 Negotiated protocol TLSv1.2
43 Negotiated cipher ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256) (limited sense as client will pick)
44 Negotiated cipher per proto (limited sense as client will pick)
45 ECDHE-RSA-AES256-SHA: TLSv1, TLSv1.1
46 ECDHE-RSA-AES256-GCM-SHA384: TLSv1.2
47 No further cipher order check has been done as order is determined by the client
48
49
50 Testing server defaults (Server Hello)
51
52 TLS extensions (standard) "extended master secret/#23" "renegotiation info/#65281" "EC point formats/#11" "encrypt-then-mac/#22" "max fragment length/#1"
53 Session Ticket RFC 5077 hint (no lifetime advertised)
54 SSL Session ID support yes
55 Session Resumption Tickets no, ID: no
56 TLS clock skew -104 sec from localtime
57 Signature Algorithm SHA256 with RSA
58 Server key size RSA 2048 bits
59 Fingerprint / Serial SHA1 A0D6FA155B1B4C3A117382FF648115DB3239801D / 03EA75BD105577AD73E8029186EFA663963D
60 SHA256 0E86D60F499B0816EB1BC3AF08BE5BCD7CFC95A7F8108AACA834B499A56BEB40
61 Common Name (CN) mx1001.wikimedia.org
62 subjectAltName (SAN) mx1001.wikimedia.org
63 Issuer Let's Encrypt Authority X3 (Let's Encrypt from US)
64 Trust (hostname) Ok via SAN and CN
65 Chain of trust Ok
66 EV cert (experimental) no
67 Certificate Expiration 88 >= 30 days (UTC: 2018-09-13 14:04 --> 2018-12-12 14:04)
68 # of certificates provided 2
69 Certificate Revocation List --
70 OCSP URI http://ocsp.int-x3.letsencrypt.org
71 OCSP stapling --
72 OCSP must staple no
73 DNS CAA RR (experimental) --
74 Certificate Transparency yes (certificate extension)
75
76
77 Testing vulnerabilities
78
79 Heartbleed (CVE-2014-0160) not vulnerable (OK), no heartbeat extension
80 CCS (CVE-2014-0224) not vulnerable (OK)
81 Ticketbleed (CVE-2016-9244), experiment. -- (applicable only for HTTPS)
82 Secure Renegotiation (CVE-2009-3555) not vulnerable (OK)
83 Secure Client-Initiated Renegotiation not vulnerable (OK)
84 CRIME, TLS (CVE-2012-4929) not vulnerable (OK) (not using HTTP anyway)
85 POODLE, SSL (CVE-2014-3566) not vulnerable (OK)
86 TLS_FALLBACK_SCSV (RFC 7507) Downgrade attack prevention supported (OK)
87 SWEET32 (CVE-2016-2183, CVE-2016-6329) VULNERABLE, uses 64 bit block ciphers
88 FREAK (CVE-2015-0204) not vulnerable (OK)
89 DROWN (CVE-2016-0800, CVE-2016-0703) not vulnerable on this host and port (OK)
90 make sure you don't use this certificate elsewhere with SSLv2 enabled services
91 https://censys.io/ipv4?q=0E86D60F499B0816EB1BC3AF08BE5BCD7CFC95A7F8108AACA834B499A56BEB40 could help you to find out
92 LOGJAM (CVE-2015-4000), experimental Common prime with 2048 bits detected: RFC5114/2048-bit DSA group with 224-bit prime order subgroup,
93 but no DH EXPORT ciphers
94 BEAST (CVE-2011-3389) TLS1: ECDHE-RSA-AES256-SHA DHE-RSA-AES256-SHA DHE-RSA-CAMELLIA256-SHA AES256-SHA CAMELLIA256-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES128-SHA DHE-RSA-CAMELLIA128-SHA AES128-SHA CAMELLIA128-SHA
95 ECDHE-RSA-DES-CBC3-SHA EDH-RSA-DES-CBC3-SHA DES-CBC3-SHA
96 VULNERABLE -- but also supports higher protocols (possible mitigation): TLSv1.1 TLSv1.2
97 LUCKY13 (CVE-2013-0169), experimental potentially VULNERABLE, uses cipher block chaining (CBC) ciphers with TLS
98 RC4 (CVE-2013-2566, CVE-2015-2808) no RC4 ciphers detected (OK)
99
100
101 Testing 359 ciphers via OpenSSL plus sockets against the server, ordered by encryption strength
102
103Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (RFC)
104-----------------------------------------------------------------------------------------------------------------------------
105 xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
106 xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
107 xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
108 x9f DHE-RSA-AES256-GCM-SHA384 DH 2048 AESGCM 256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
109 xcca8 ECDHE-RSA-CHACHA20-POLY1305 ECDH 256 ChaCha20 256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
110 xccaa DHE-RSA-CHACHA20-POLY1305 DH 2048 ChaCha20 256 TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256
111 xc09f DHE-RSA-AES256-CCM DH 2048 AESCCM 256 TLS_DHE_RSA_WITH_AES_256_CCM
112 x6b DHE-RSA-AES256-SHA256 DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
113 x39 DHE-RSA-AES256-SHA DH 2048 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
114 xc077 ECDHE-RSA-CAMELLIA256-SHA384 ECDH 256 Camellia 256 TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384
115 xc4 DHE-RSA-CAMELLIA256-SHA256 DH 2048 Camellia 256 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256
116 x88 DHE-RSA-CAMELLIA256-SHA DH 2048 Camellia 256 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
117 x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384
118 xc09d AES256-CCM RSA AESCCM 256 TLS_RSA_WITH_AES_256_CCM
119 x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256
120 x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA
121 xc0 CAMELLIA256-SHA256 RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256
122 x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
123 xc07b - RSA CamelliaGCM 256 TLS_RSA_WITH_CAMELLIA_256_GCM_SHA384
124 xc07d - DH 2048 CamelliaGCM 256 TLS_DHE_RSA_WITH_CAMELLIA_256_GCM_SHA384
125 xc08b - ECDH 521 CamelliaGCM 256 TLS_ECDHE_RSA_WITH_CAMELLIA_256_GCM_SHA384
126 xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
127 xc027 ECDHE-RSA-AES128-SHA256 ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
128 xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
129 x9e DHE-RSA-AES128-GCM-SHA256 DH 2048 AESGCM 128 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
130 xc09e DHE-RSA-AES128-CCM DH 2048 AESCCM 128 TLS_DHE_RSA_WITH_AES_128_CCM
131 xc09c AES128-CCM RSA AESCCM 128 TLS_RSA_WITH_AES_128_CCM
132 x67 DHE-RSA-AES128-SHA256 DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
133 x33 DHE-RSA-AES128-SHA DH 2048 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
134 xc076 ECDHE-RSA-CAMELLIA128-SHA256 ECDH 256 Camellia 128 TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256
135 xbe DHE-RSA-CAMELLIA128-SHA256 DH 2048 Camellia 128 TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256
136 x45 DHE-RSA-CAMELLIA128-SHA DH 2048 Camellia 128 TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
137 x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256
138 x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256
139 x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA
140 xba CAMELLIA128-SHA256 RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256
141 x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
142 xc07a - RSA CamelliaGCM 128 TLS_RSA_WITH_CAMELLIA_128_GCM_SHA256
143 xc07c - DH 2048 CamelliaGCM 128 TLS_DHE_RSA_WITH_CAMELLIA_128_GCM_SHA256
144 xc08a - ECDH 521 CamelliaGCM 128 TLS_ECDHE_RSA_WITH_CAMELLIA_128_GCM_SHA256
145 xc012 ECDHE-RSA-DES-CBC3-SHA ECDH 521 3DES 168 TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
146 x16 EDH-RSA-DES-CBC3-SHA DH 2048 3DES 168 TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
147 x0a DES-CBC3-SHA RSA 3DES 168 TLS_RSA_WITH_3DES_EDE_CBC_SHA
148
149
150 Running client simulations via sockets
151
152 Java 6u45 TLSv1.0 AES128-SHA
153 Java 7u25 TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
154 Java 8u31 TLSv1.2 ECDHE-RSA-AES128-SHA256, 256 bit ECDH (P-256)
155 OpenSSL 1.0.1l TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 521 bit ECDH (P-521)
156 OpenSSL 1.0.2e TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
157
158 Done 2018-09-14 15:01:40 [ 156s] -->> 208.80.154.76:25 (mx1001.wikimedia.org) <<--

In https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/458061/ @MoritzMuehlenhoff mentioned gathering additional stats before making changes which makes a lot of sense. So Exim TLS ciphersuite metrics were implemented and a breakdown has been added to the mail Grafana dashboard for review:

https://grafana.wikimedia.org/dashboard/db/mail?refresh=5m&panelId=31&fullscreen&orgId=1&from=1536339405902&to=1536944205902&var-datasource=eqiad%20prometheus%2Fops

In terms of next steps I propose enabling %SERVER_PRECEDENCE in exim gnutls preferences, monitor the distribution for a few days, then disable TLS1.0. Thoughts?

https://gnutls.org/manual/html_node/Priority-Strings.html
%SERVER_PRECEDENCE The ciphersuite will be selected according to server priorities and not the client’s

Change 460961 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] mx: enable gnutls %SERVER_PRECEDENCE in exim

https://gerrit.wikimedia.org/r/460961

[operations/puppet@production] mx: enable gnutls %SERVER_PRECEDENCE in exim

I'll plan to merge the above tomorrow barring objections

Change 460961 merged by Herron:
[operations/puppet@production] mx: enable gnutls %SERVER_PRECEDENCE in exim

https://gerrit.wikimedia.org/r/460961

Reworked and added a few more panels to the bottom of https://grafana.wikimedia.org/dashboard/db/mail to better show the distribution of ciphers and TLS versions.

Aklapper added a subscriber: herron.

Removing task assignee due to inactivity, as this open task has been assigned to the same person for more than two years (see the emails sent to the task assignee on Oct27 and Nov23). Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome.
(See https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator.)