Page MenuHomePhabricator

Letsencrypt all the prod things we can - planning
Closed, ResolvedPublic

Description

Now that we have basic puppetization of LE certs, we can start planning to deploy them in some limited places in production where we have existing commercial certs on renewal cycles.

Keep in mind the following situations are off the table (for now, here):

  1. This isn't about the junk redirect domains, that's in T133548
  2. If it's not a singular, direct, public-facing HTTP[S] host, it's out for now due to puppetization limitations
  3. 3rd party services/hosts are out, and I'm not looking at fundraising or labs specifically in this ticket either
  4. Wildcard certs (LE doesn't support)

Looking at our Ops cert renewal calendar and our globalsign account (where most certs come from), and keeping the above restrictions in mind, these seem to be the applicable cases we could LE-ify, ordered by their next commercial cert expiry:

CertExpirySwitched to LE
icinga.wm.o2017-02-06Yes
ganglia.wm.o2017-02-07Yes
librenms.wm.o2017-02-10Yes
wikitech.wm.o2017-02-23Yes
lists.wm.o2017-03-01Yes
tendril.wm.o2017-03-17Yes
dumps.wm.o2017-04-26Yes
archiva.wm.o2017-05-08Yes
gerrit.wm.o2018-05-25Yes
wikitech-static.w.o2017-03-01Yes
mx[12]001.wm.o2017-09-22Yes

I just turned on our first full-auto-puppeted LE certs today (for apt/ubuntu/mirrors, which had no certs at all before), which is an nginx host. I'm going test one other from the above list which is an Apache.

We can put off mass conversion of this list for a while (they don't start expiring until 2017 anyways) until we've had some history on the first hosts and are comfortable the LE situation is stable with auto-renewals and all that. So we're probably looking at ~60+ days out from now (circa late June or beyond) before we start converting the remaining services above to LE certs.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a project: Operations. · View Herald TranscriptApr 26 2016, 5:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Southparkfan updated the task description. (Show Details)Apr 26 2016, 5:57 PM
Southparkfan added a subscriber: Southparkfan.

Change 285441 had a related patch set uploaded (by BBlack):
ganglia: use LE cert

https://gerrit.wikimedia.org/r/285441

Change 285442 had a related patch set uploaded (by BBlack):
ganglia: remove old cert absent line

https://gerrit.wikimedia.org/r/285442

BBlack updated the task description. (Show Details)Apr 26 2016, 8:08 PM

Change 285572 had a related patch set uploaded (by BBlack):
rt.wm.o: use LE cert

https://gerrit.wikimedia.org/r/285572

Change 285573 had a related patch set uploaded (by BBlack):
rt.wm.o: remove old cert definition

https://gerrit.wikimedia.org/r/285573

Change 285441 abandoned by BBlack:
ganglia: use LE cert

Reason:
using rt for LE apache test instead

https://gerrit.wikimedia.org/r/285441

Change 285442 abandoned by BBlack:
ganglia: remove old cert absent line

Reason:
using rt for LE apache test instead

https://gerrit.wikimedia.org/r/285442

Change 285572 merged by BBlack:
rt.wm.o: use LE cert

https://gerrit.wikimedia.org/r/285572

BBlack updated the task description. (Show Details)Apr 27 2016, 3:30 AM

Table at top updated. rt.wikimedia.org is on LE now as our first example with Apache.

Change 285573 merged by BBlack:
rt.wm.o: remove old cert definition

https://gerrit.wikimedia.org/r/285573

fgiunchedi triaged this task as Normal priority.Apr 27 2016, 9:25 AM
BBlack updated the task description. (Show Details)Aug 15 2016, 3:35 PM
BBlack moved this task from Triage to TLS on the Traffic board.Sep 30 2016, 1:40 PM

Change 330633 had a related patch set uploaded (by Dzahn):
icinga: use Letsencrypt for SSL cert, spend less donor money on prime numbers

https://gerrit.wikimedia.org/r/330633

Dzahn added a subscriber: Dzahn.Jan 5 2017, 6:23 AM

Change 330829 had a related patch set uploaded (by Dzahn):
tendril: use Letsencrypt for SSL cert

https://gerrit.wikimedia.org/r/330829

Change 330633 merged by Dzahn:
icinga: use Letsencrypt for SSL cert, spend less donor money on prime numbers

https://gerrit.wikimedia.org/r/330633

Change 330841 had a related patch set uploaded (by Dzahn):
icinga: Include challenge-apache.conf, exclude acme from proto redirect

https://gerrit.wikimedia.org/r/330841

Change 330841 merged by Dzahn:
icinga: Include challenge-apache.conf, exclude acme from proto redirect

https://gerrit.wikimedia.org/r/330841

Dzahn updated the task description. (Show Details)Jan 6 2017, 4:18 AM

Icinga switched to LE just now.

Change 331085 had a related patch set uploaded (by Dzahn):
ganglia: use Letsencrypt for SSL cert

https://gerrit.wikimedia.org/r/331085

RobH mentioned this in Unknown Object (Task).Jan 9 2017, 8:36 PM
RobH updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)Jan 9 2017, 8:51 PM
RobH updated the task description. (Show Details)Jan 9 2017, 11:41 PM

Change 330829 merged by Dzahn:
tendril: use Letsencrypt for SSL cert

https://gerrit.wikimedia.org/r/330829

Dzahn updated the task description. (Show Details)Jan 17 2017, 6:53 PM
RobH updated the task description. (Show Details)Jan 25 2017, 7:58 PM

Seems wikitech-static was converted previously, so it was already done.

RobH updated the task description. (Show Details)Jan 26 2017, 1:29 AM
RobH updated the task description. (Show Details)

This likely shouldn't close yet, and we should add in mx/mail systems.

Dzahn added a subscriber: hashar.Jan 27 2017, 4:56 AM

@hashar which CI systems had SSL certs again please

@Dzahn I should have written down somewhere following our conversation from last week or so. For the CI we have the following domains all serving HTTP being force redirected to HTTPS:

Legacy ones under mediawiki.org. they point to the TEXT load balancer which handles the redirection:

  • doc.mediawiki.org
  • integration.mediawiki.org

At https level, both use the star cert from DigiCert:

  • Server certificate: *.wikipedia.org
  • Server certificate: DigiCert SHA2 High Assurance Server CA
  • Server certificate: DigiCert High Assurance EV Root CA

Then the canonical entries are under wikimedia.org, same hostnames but they point to the MISC load balancer which route the requests to contint1001.

  • doc.wikimedia.org
  • integration.wikimedia.org
  • Server certificate: *.wikipedia.org
  • Server certificate: DigiCert SHA2 High Assurance Server CA
  • Server certificate: DigiCert High Assurance EV Root CA

Summary is:

  • *.MediaWiki.org --> text LB and star certificate
  • *.WikiMedia.org --> misc LB and star certificate
hashar updated the task description. (Show Details)Jan 27 2017, 9:43 AM
Dzahn added a comment.Jan 27 2017, 2:33 PM

@hashar thank you for this very detailed reply. Since everything is already behind varnish i think it will not be relevant in the context of this ticket then because of "2)" in the ticket description. I was checking if maybe we had some self-signed certs for inter-service communication on contint1001/2001 left or something.

I don't think we ever used self-signed certs for CI. Internal communications I can remember of are:

So I guess there is no need for LetsEncrypt as long as CI is behind misc-web and the star certificate. Thx Daniel :-}

RobH awarded a token.Jan 27 2017, 5:55 PM
hashar removed a subscriber: hashar.Mar 1 2017, 10:30 PM
grin added a subscriber: grin.Apr 7 2017, 8:13 PM

Just as a sidenote: be aware that wildcards are only wildcard one level up, not any; *.wikimedia.org matches robh.wikimedia.org but not server01.robh.wikimedia.org (which became obvious on the OSM tileservers on labs).

alex@alex-laptop:~$ openssl s_client -starttls smtp -connect mx1001.wikimedia.org:25 2>/dev/null | openssl x509 -noout -text | grep Issuer:
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
^C
alex@alex-laptop:~$ openssl s_client -starttls smtp -connect mx2001.wikimedia.org:25 2>/dev/null | openssl x509 -noout -text | grep Issuer:
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
^C

So I think the description can be updated.

And actually that's the whole list on this ticket. Anything else missing @BBlack or can this be closed?

Yeah I think this is closeable. This was just our initial "convert all the low-hanging fruit" ticket for the previous iteration of LE support.

Krenair closed this task as Resolved.Aug 28 2018, 5:11 PM
Krenair updated the task description. (Show Details)

Yep cool