Page MenuHomePhabricator
Create Task
Maniphest T256806

Mailserver TLS is broken, root certificates are not present for sent intermediate certificates
Closed, ResolvedPublic
Actions

Description

openssl s_client -connect mail.tools.wmflabs.org:25 -starttls smtp outputs:

CONNECTED(00000003)
depth=0 CN = mail.tools.wmflabs.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = mail.tools.wmflabs.org
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=mail.tools.wmflabs.org
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEmjCCA4KgAwIBAgISA3CQV+TEuTEUJdFrfPfEXdMmMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDA2MjMxNTA2MzNaFw0y
MDA5MjExNTA2MzNaMCExHzAdBgNVBAMTFm1haWwudG9vbHMud21mbGFicy5vcmcw
WTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATZLb0lH6NvZWRuhZS3h4Fh8HuxgDhp
3FaiAuj/o1/jv9fCk+yjIW0gNus0L8Y1Hrvufz8nJb/Tt8LaP3eKv0BHo4ICbDCC
AmgwDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcD
AjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTlsZoULnb33wrco/85KhEF0WTGlDAf
BgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggrBgEFBQcBAQRjMGEw
LgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRzZW5jcnlwdC5vcmcw
LwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5jcnlwdC5vcmcv
MCEGA1UdEQQaMBiCFm1haWwudG9vbHMud21mbGFicy5vcmcwTAYDVR0gBEUwQzAI
BgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nw
cy5sZXRzZW5jcnlwdC5vcmcwggEFBgorBgEEAdZ5AgQCBIH2BIHzAPEAdgDwlaRZ
8gDRgkAQLS+TiI6tS/4dR+OZ4dA0prCoqo6ycwAAAXLh7lrIAAAEAwBHMEUCIFDh
jsTfCm1ILSgi3ZKnuFms/lmOi+sSta5cfUBQSaYFAiEAlcjjAgWXCVX8PSe9Mx9s
EZ6n2mNnEjrUdR0Zs5zRiSAAdwAHt1wb5X1o//Gwxh0jFce65ld8V5S3au68YToa
adOiHAAAAXLh7lr3AAAEAwBIMEYCIQDdZMb0rDjUR68a2wELpn6t78mSPXDZ5NGD
Nv0gFvZtmAIhANo35YavWGskBp+vwkHcaTDbIpdJATtsliiiGwpkdNi6MA0GCSqG
SIb3DQEBCwUAA4IBAQATO5KBp110TjvVtoWoumkPhPZjNb60ygZ5nChZ1e2FY1+L
UAEw7k2r/hKWYbD4cfXZ6dEJtzlSVDWvXpej0pD+R3GWbrJFkQAESA2wQFACnfh1
FmAvI3+pHEPoKv+RCRrxoUGFcL+Nc3MloBABe6mbXkrtc1Qm40rxZ5ZW8kMZ5lrR
I3E7cCx6NTOG9+NLLHHLM+5CR81kx5lnrQC6SpTmDOSFXhNialG1piE1smqgjBVI
9eePHLLVh5hzYqK/Ih7z77qA3YeJFNI1o/odnq5I+x188kYWfhNVe1Dzzvv1RG1Y
exDMvI1TjSWCODL6idZujOO/Tu1lcK/wTEIMczkC
-----END CERTIFICATE-----
subject=/CN=mail.tools.wmflabs.org
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1721 bytes and written 335 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
    Session-ID: 6E1DB07FFC3974A597BDA6C942EE025BEF2E7D6759C2E020C23C0049FA75A220
    Session-ID-ctx: 
    Master-Key: A072F47C634E9B4631FF592B90E72F691DE5D32A6E2309AF9B5044FBDE22A7B45957E30CEF208B3EAD39ED0B6A456A20
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1593547835
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes
---
250 HELP

The certificate chain is ending early, at the intermediate certificate, and not landing correctly at the Let's Encrypt root like it should. This means any client trying to send mail with STARTTLS will fail.

Details

SubjectRepoBranchLines +/-
toolforge: Use chained cert for mail relay TLSoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Naypta created this task.Jun 30 2020, 8:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2020, 8:14 PM
Naypta renamed this task from Mailserver TLS is broken, intermediate certificates are not present to Mailserver TLS is broken, root certificates are not present for sent intermediate certificates.Jun 30 2020, 8:14 PM
Naypta added a parent task: T249237: Fix Cloud VPS and Toolforge mail servers to work with the modern internet.
Krenair subscribed.Jun 30 2020, 8:15 PM

This is probably because this:
modules/profile/templates/toolforge/mail-relay.exim4.conf.erb:tls_certificate = /etc/acmecerts/<%= @cert_name %>/live/ec-prime256v1.crt
should use .chained.crt more like these:

modules/profile/templates/exim/exim4.conf.smarthost.erb:tls_certificate = /etc/acme/cert/<%= @cert_name.gsub(/[-.]/, '_') %>.chained.crt
modules/profile/templates/exim/exim4.conf.mailman.erb:tls_certificate = /etc/acmecerts/lists/live/rsa-2048.chained.crt
modules/role/templates/exim/exim4.conf.mx.erb:tls_certificate = /etc/acmecerts/mx/live/rsa-2048.chained.crt
gerritbot added a comment.Jun 30 2020, 8:16 PM

Change 608720 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolforge: Use chained cert for mail relay TLS

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608720

gerritbot added a project: Patch-For-Review.Jun 30 2020, 8:16 PM
bd808 claimed this task.Jun 30 2020, 8:20 PM
bd808 triaged this task as High priority.
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
bd808 added a subscriber: aborrero.
gerritbot added a comment.Jul 1 2020, 9:19 AM

Change 608720 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: Use chained cert for mail relay TLS

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608720

aborrero closed this task as Resolved.Jul 1 2020, 9:26 AM

Thanks for your report @Naypta !

I merged @bd808 patch (thanks!!) and things look better now:

arturo@endurance:~$ openssl s_client -connect mail.tools.wmflabs.org:25 -starttls smtp
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = mail.tools.wmflabs.org
verify return:1
---
Certificate chain
 0 s:CN = mail.tools.wmflabs.org
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEmjCCA4KgAwIBAgISA3CQV+TEuTEUJdFrfPfEXdMmMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDA2MjMxNTA2MzNaFw0y
MDA5MjExNTA2MzNaMCExHzAdBgNVBAMTFm1haWwudG9vbHMud21mbGFicy5vcmcw
WTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATZLb0lH6NvZWRuhZS3h4Fh8HuxgDhp
3FaiAuj/o1/jv9fCk+yjIW0gNus0L8Y1Hrvufz8nJb/Tt8LaP3eKv0BHo4ICbDCC
AmgwDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcD
AjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTlsZoULnb33wrco/85KhEF0WTGlDAf
BgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggrBgEFBQcBAQRjMGEw
LgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRzZW5jcnlwdC5vcmcw
LwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5jcnlwdC5vcmcv
MCEGA1UdEQQaMBiCFm1haWwudG9vbHMud21mbGFicy5vcmcwTAYDVR0gBEUwQzAI
BgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nw
cy5sZXRzZW5jcnlwdC5vcmcwggEFBgorBgEEAdZ5AgQCBIH2BIHzAPEAdgDwlaRZ
8gDRgkAQLS+TiI6tS/4dR+OZ4dA0prCoqo6ycwAAAXLh7lrIAAAEAwBHMEUCIFDh
jsTfCm1ILSgi3ZKnuFms/lmOi+sSta5cfUBQSaYFAiEAlcjjAgWXCVX8PSe9Mx9s
EZ6n2mNnEjrUdR0Zs5zRiSAAdwAHt1wb5X1o//Gwxh0jFce65ld8V5S3au68YToa
adOiHAAAAXLh7lr3AAAEAwBIMEYCIQDdZMb0rDjUR68a2wELpn6t78mSPXDZ5NGD
Nv0gFvZtmAIhANo35YavWGskBp+vwkHcaTDbIpdJATtsliiiGwpkdNi6MA0GCSqG
SIb3DQEBCwUAA4IBAQATO5KBp110TjvVtoWoumkPhPZjNb60ygZ5nChZ1e2FY1+L
UAEw7k2r/hKWYbD4cfXZ6dEJtzlSVDWvXpej0pD+R3GWbrJFkQAESA2wQFACnfh1
FmAvI3+pHEPoKv+RCRrxoUGFcL+Nc3MloBABe6mbXkrtc1Qm40rxZ5ZW8kMZ5lrR
I3E7cCx6NTOG9+NLLHHLM+5CR81kx5lnrQC6SpTmDOSFXhNialG1piE1smqgjBVI
9eePHLLVh5hzYqK/Ih7z77qA3YeJFNI1o/odnq5I+x188kYWfhNVe1Dzzvv1RG1Y
exDMvI1TjSWCODL6idZujOO/Tu1lcK/wTEIMczkC
-----END CERTIFICATE-----
subject=CN = mail.tools.wmflabs.org

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2896 bytes and written 473 bytes
Verification: OK
---
[...]

I'm closing this task now, please feel free to reopen if required.

Maintenance_bot removed a project: Patch-For-Review.Jul 1 2020, 8:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits