mail.wikimedia.org SSL cert expiring Mon 23 Oct 2017
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	herron
	Aug 24 2017, 7:44 PM

Description

Icinga has open check_smtp warnings for mx1001 and mx2001

certificate 'mail.wikimedia.org' expires in 59 day(s) (Mon 23 Oct 2017 06:01:00 PM UTC)

Opening task to investigate renewal

Details

Subject	Repo	Branch	Lines +/-
mx: remove sslcert::certificate for mail.wikimedia.org	operations/puppet	production	+0 -6
mx: get rid of the old mail.wikimedia.org certificate	operations/puppet	production	+1 -0
MX: Change Exim configuration to use letsencrypt certificate	operations/puppet	production	+2 -2
Add letsencrypt certs to mx servers	operations/puppet	production	+20 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T133717 Letsencrypt all the prod things we can - planning
Resolved	RobH	T159346 convert mail servers from GS to LE certificates
Resolved	herron	T174081 mail.wikimedia.org SSL cert expiring Mon 23 Oct 2017
Resolved	herron	T174720 letsencrypt::cert::integrated and non-http servers
Resolved	fgiunchedi	T181519 tls expiry check for mx vs acme-setup renewal period

Event Timeline

herron created this task.Aug 24 2017, 7:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 24 2017, 7:44 PM

Looking into how to renew this using let's encrypt. The globalsign cert used today is configured with attributes:

CN=mail.wikimedia.org
SAN=cert mail.wikimedia.org, mx1001.wikimedia.org, mx1002.wikimedia.org, mx2001.wikimedia.org, mx2002.wikimedia.org

However, mail.wikimedia.org, mx1002.wikimedia.org and mx2002.wikimedia.org no longer exist in DNS:

Host mail.wikimedia.org not found: 3(NXDOMAIN)
Host mx1002.wikimedia.org not found: 3(NXDOMAIN)
Host mx2002.wikimedia.org not found: 3(NXDOMAIN)

This leaves only mx1001.wikimedia.org and mx2001.wikimedia.org.

This being the case it looks like by switching from a "shared" SAN cert to individual letscencrypt certs we could deploy replacements using letsencrypt::cert::integrated along with a small httpd to handle validation requests to the fqdn of each mx.

For the history side of it :), mx1002/mx2002 never existed, it was just me hoping to get around in building additional MXes (and possibly splitting roles, e.g. inbound and outbound) and since adding SANs later costs, I just added them there to be on the safe side. As for mail.wikimedia.org... these was just a made-up subject to avoid picking one out of four hostnames/SANs as subject.

Now that we have Let's Encrypt and free certs, switching to individual, separate certificates per host, each with their hostname as subject like @herron proposed totally makes sense to me.

herron added a subtask: T174720: letsencrypt::cert::integrated and non-http servers.Aug 31 2017, 8:04 PM

Krenair subscribed.Aug 31 2017, 9:28 PM

Change 375427 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] WIP: Add letsencrypt certs to mx servers

https://gerrit.wikimedia.org/r/375427

gerritbot added a project: Patch-For-Review.Sep 1 2017, 7:39 PM

herron mentioned this in T174720: letsencrypt::cert::integrated and non-http servers.Sep 19 2017, 7:09 PM

Change 375427 merged by Herron:
[operations/puppet@production] Add letsencrypt certs to mx servers

https://gerrit.wikimedia.org/r/375427

Change 384591 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] MX: Change Exim configuration to use letsencrypt certificate

https://gerrit.wikimedia.org/r/384591

Change 384591 merged by Herron:
[operations/puppet@production] MX: Change Exim configuration to use letsencrypt certificate

https://gerrit.wikimedia.org/r/384591

LE certs have been deployed to mx1001 and mx2001.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:f0:ef:69:c4:af:f9:b1:82:f4:57:31:9d:42:97:2b:a2:54
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Oct 11 13:05:38 2017 GMT
            Not After : Jan  9 13:05:38 2018 GMT
        Subject: CN=mx1001.wikimedia.org

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:17:49:62:3d:78:eb:a6:d6:d2:9a:d7:36:ca:dd:da:48:0b
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Oct 11 12:11:07 2017 GMT
            Not After : Jan  9 12:11:07 2018 GMT
        Subject: CN=mx2001.wikimedia.org

fgiunchedi created subtask T181519: tls expiry check for mx vs acme-setup renewal period.Nov 28 2017, 4:04 PM

herron added a parent task: T159346: convert mail servers from GS to LE certificates.Jan 25 2018, 2:31 AM

herron mentioned this in T159346: convert mail servers from GS to LE certificates.Jan 25 2018, 2:36 AM

fgiunchedi closed subtask T181519: tls expiry check for mx vs acme-setup renewal period as Resolved.Feb 19 2018, 9:25 AM

Krenair closed subtask T174720: letsencrypt::cert::integrated and non-http servers as Resolved.Aug 28 2018, 4:56 PM

Change 478964 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] mx: get rid of the old mail.wikimedia.org certificate

https://gerrit.wikimedia.org/r/478964

Change 478965 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] mx: remove sslcert::certificate for mail.wikimedia.org

https://gerrit.wikimedia.org/r/478965

Change 478964 merged by Vgutierrez:
[operations/puppet@production] mx: get rid of the old mail.wikimedia.org certificate