Page MenuHomePhabricator

TLS not working on mx-out0[12].wmflabs.org
Closed, ResolvedPublic

Description

Found while debugging for T212709: discourse-mediawiki.wmflabs.org and discourse.wmflabs.org send emails only to @wikimedia.org addresses:

$ ssh mx-out01.wmflabs.org
$ grep tls_certificate /etc/exim4/exim4.conf
tls_certificate = /etc/acme/cert/mx-out01.cloudinfra.wmflabs.org.chained.crt
$ openssl x509 -in /etc/acme/cert/mx-out01.cloudinfra.wmflabs.org.chained.crt -text -noout
x509: Cannot open input file /etc/acme/cert/mx-out01.cloudinfra.wmflabs.org.chained.crt, No such file or directory
x509: Use -help for summary.
$ ls /etc/acme/cert
mx_out01.chain.crt
mx_out01.chained.crt
mx_out01_cloudinfra_wmflabs_org.chain.crt
mx_out01_cloudinfra_wmflabs_org.chained.crt
mx_out01_cloudinfra_wmflabs_org.crt
mx_out01.crt
$ ssh mx-out02.wmflabs.org
$ grep tls_certificate /etc/exim4/exim4.conf
tls_certificate = /etc/acme/cert/mx-out02.wmflabs.org.chained.crt
$ openssl x509 -in /etc/acme/cert/mx-out02.wmflabs.org.chained.crt -text -noout
x509: Cannot open input file /etc/acme/cert/mx-out02.wmflabs.org.chained.crt, No such file or directory
x509: Use -help for summary.
$ ls /etc/acme/cert
mx_out02_cloudinfra_wmflabs_org.chain.crt    mx_out02_wmflabs_org.chain.crt
mx_out02_cloudinfra_wmflabs_org.chained.crt  mx_out02_wmflabs_org.chained.crt
mx_out02_cloudinfra_wmflabs_org.crt          mx_out02_wmflabs_org.crt

I'm not sure if the fix is just adjusting the profile::mail::smarthost::cert_subjects and profile::mail::smarthost::cert_name values for each server or something deeper.

Event Timeline

Change 482113 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] mail::smarthost: exim4: replace '-' and '.' in cert names with '_'

https://gerrit.wikimedia.org/r/482113

Change 482113 merged by Herron:
[operations/puppet@production] mail::smarthost: exim4: replace '-' and '.' in cert names with '_'

https://gerrit.wikimedia.org/r/482113

This has been corrected, and can confirm that TLS is working on mx-out0[12]

herron@mx-out01:~$ openssl s_client -connect mx-out01.wmflabs.org:25 -starttls smtp
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = mx-out01.wmflabs.org
verify return:1
---
Certificate chain
 0 s:/CN=mx-out01.wmflabs.org
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFXzCCBEegAwIBAgISA/rU0v9mmEMIFnFGoe9AucbnMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xOTAxMDMxNzUxMjJaFw0x
OTA0MDMxNzUxMjJaMB8xHTAbBgNVBAMTFG14LW91dDAxLndtZmxhYnMub3JnMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyaejVY5pxfO/xtFxCqQYrTnR
hKxMyGyTQUuX2MHnsxcjRg/1Nx1zuDk5/wOyLm001y6UI7NXGlWD1Z9xaCsCglyW
RDFWlIu+km9BRzrAgcZgqKF/Yfifa70nPpBGlQbFnyCOziHrn2Ev2nxbd4yYyHgx
VyGDEUh5Dhpxo8Vn+RRk5hJbIxPA1kbFKN2fiX+LtkP3Y267gZ9oRg6E9ok8dzn1
0F+INwvCy6NlBqm4GDcKqXJ1yaFYYkkcszeQb19Zxt/u53l/MIcLWN3eYYMNW3qe
i2H1B7ZNeX28MGfdF6WtLV0cskgjGfgHJQZuT7rnpRsIsJkvyuXW11Qd6FASsQID
AQABo4ICaDCCAmQwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMB
BggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBRU8qjKXRUFEUKKzPSj
RdikfLsXgjAfBgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggrBgEF
BQcBAQRjMGEwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRzZW5j
cnlwdC5vcmcwLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5j
cnlwdC5vcmcvMB8GA1UdEQQYMBaCFG14LW91dDAxLndtZmxhYnMub3JnMEwGA1Ud
IARFMEMwCAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0
dHA6Ly9jcHMubGV0c2VuY3J5cHQub3JnMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDv
AHYAdH7agzGtMxCRIZzOJU9CcMK//V5CIAjGNzV55hB7zFYAAAFoFQ5CcwAABAMA
RzBFAiAu4M0JbhAtE5yUH/FfALyloraF/wquUPkHC651JfRxIQIhAMF/PbTLc2SH
p8KTVW1L3umnQXDe7uZmN8Av2CoCSb4FAHUAKTxRllTIOWW6qlD8WAfUt2+/WHop
ctykwwz05UVH9HgAAAFoFQ5E1AAABAMARjBEAiBJJTH4CK6dXMKKmIDrrwv+QtVS
ELgVvI7uX6Q+tT5XgQIgKujpZVCQWJfxDsQN+AXjpViCjSdZgWPAvxrtGoyFiuUw
DQYJKoZIhvcNAQELBQADggEBACtP2nqrx2j9qKuEtXjma8zzAnCCwxtlTlZxjFUC
j9rPfIPb/Xg5BbBPuJH18jFDn1GVNyQpcfbQYeACGRB44I7R5VZa591xeKP5l3WX
Qv/wTGwcAqWA0aHxgn7bwvPK52qjelQ2EaUdi9hZoLg8RPiSiGujrp9/XKl7/oNv
aoX4XAVHSujUl1n0Sh6kXColPaNnrCF70b/gQh68QjhPkhlzVE05xwcHag8+eIDX
YfxWgoS2okj8+GLcM0oAFFhyNlElip9/DaC5DLmm0rwA3huDPO+AWU0/OpyjKPCZ
X4bxPV4RHgb/X2JDTGscpqG4TWCxlFmOVAECl70wswD5soo=
-----END CERTIFICATE-----
subject=/CN=mx-out01.wmflabs.org
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3311 bytes and written 335 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: C27B6D7CA8569A5AD6BD780BB7352FDEDB143D7E3705E6F2039BF4607D9F8CED
    Session-ID-ctx: 
    Master-Key: B60B9F6EA1315ACA7A4B9570FF27234C66E28914CCAC4E3E057EAA0F98A46EB1BAC3A2DAB650D0623F969123D5F5D8DA
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1546542235
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
250 HELP
QUIT
DONE
herron@mx-out01:~$ openssl s_client -connect mx-out02.wmflabs.org:25 -starttls smtp
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = mx-out02.wmflabs.org
verify return:1
---
Certificate chain
 0 s:/CN=mx-out02.wmflabs.org
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgISA5TZwzPzI7Vmo79laGuaFcLEMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODEyMDQyMzQ1MzlaFw0x
OTAzMDQyMzQ1MzlaMB8xHTAbBgNVBAMTFG14LW91dDAyLndtZmxhYnMub3JnMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsxdq3WBT5TaxG26F6qj+OinH
1YZvVDqLAiDG50acN69Wy1kVmTSdVF1AXKkADBou9nXLzGHkfoTJAyYjKlRenrTm
Z21C2JwWexIr+/jNIOmdfUwgL2JjGN9RU4zxXvUgQR+MM2QZh4yOnC0L40FoEP8q
Fu27Ye3/Up9oU0x/sj8Yy7QieQxUr/vV9tgtUYvjesUywLV6Sg+wA2+iYw1/E3wO
f/IDKEnS6YJ2ksibVZ/atcpy4MoTyhE0RBDV1IfAIoQQe+ZeOYPVlKo6sTSGdf8W
137l+8h5WB2gF++0hqDBKfn2XInqaIS0AMW8NV+0RL3R1V5P6fQGcGp7aZfF3QID
AQABo4ICaTCCAmUwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMB
BggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTnH8+I3I9ZMjONr5tH
3SzX+8VpLTAfBgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggrBgEF
BQcBAQRjMGEwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRzZW5j
cnlwdC5vcmcwLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5j
cnlwdC5vcmcvMB8GA1UdEQQYMBaCFG14LW91dDAyLndtZmxhYnMub3JnMEwGA1Ud
IARFMEMwCAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0
dHA6Ly9jcHMubGV0c2VuY3J5cHQub3JnMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDw
AHYA4mlLribo6UAJ6IYbtjuD1D7n/nSI+6SPKJMBnd3x2/4AAAFne9PV+AAABAMA
RzBFAiB8gD8H4wT32hR1IiO505tmS+usOF3B71soBcHgQ00aFgIhAOIx6A6Jz/IG
NwC00Vhuya2b9j13rZi8PVNlMdqmJGccAHYAKTxRllTIOWW6qlD8WAfUt2+/WHop
ctykwwz05UVH9HgAAAFne9PWQgAABAMARzBFAiEA7YPd7gBEb27rG7P+8jMHE99A
DEHZWhaBlw2KbfjEuJ0CICF7oZHtiVMK4HTlLO61RDeRsTRM0P+K4fsAzRdviQlq
MA0GCSqGSIb3DQEBCwUAA4IBAQARhW77SzjHoMY8XI4rmHxRXqidodKnNlQ1b+1h
JLzGNaSKaYQhVoMLwHzqAXNcDKdfaMY6tssyVbUfzc5XEsfRizY5AxUVRDbqGTuJ
DwuH07LP+clajK4R2E59so+c5HrtVOVoSlCwG2YoPOGF1EVm9pdK4Fffxlcj4pZX
pWHsay4Uyc9fL3fqK0y6YDtE/zgRzqp95Kn19WyZk0UvHnXvPPSss4IHiERRHChu
3YkqCtyhH1KzrE/YGJLHzGeo9n/lEpONxsEQfsRqLyJWxgBSA9tDbuEzroEy18v8
H6QfP/ANCSK0/0PZl/cNK3YoVv9NtgidV2SFuCaJAQZh/4DM
-----END CERTIFICATE-----
subject=/CN=mx-out02.wmflabs.org
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3312 bytes and written 335 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: EDF29C341781F09A8EDD705A5047EEB772DFA057C39FE630A78C4BE045192286
    Session-ID-ctx: 
    Master-Key: DD94D34D3AC6CEDC61A7D8E23CD862922C78E8B863FF0A6AAF1FDC35054DB624FCB773885726DB43B045FCCB4234440D
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1546542252
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
250 HELP
QUIT
DONE