Page MenuHomePhabricator
Create Task
Maniphest T182927

Get letsencrypt wildcard cert for *.beta.wmflabs.org domains
Closed, ResolvedPublic
Actions

Description

Original title: various .beta.wmflabs.org domains use an invalid ssl certificate

There are quite a few domains that are present in DNS, but send an invalid ssl certificate and thus create browser warnings. So far I've found that for all projects ("projects" like in wikipedia/wikibooks/... that is), m.$project.beta..., zero.$project.beta... and $project.beta... have that problem.

Details

SubjectRepoBranchLines +/-
profile::cache::ssl::unified: Allow passing certs/certs_active by hieraoperations/puppetproduction+20 -10
Allow acme-chief to provide unified certoperations/puppetproduction+47 -7
tlsproxy: Ensure OCSP stapling nginx reload hook present for acme-chiefoperations/puppetproduction+1 -0
openstack: Permit deployment-prep-dns-manager to log in from instance subnetoperations/puppetproduction+2 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

EddieGP created this task.Dec 15 2017, 12:12 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 15 2017, 12:12 AM
Dzahn subscribed.Dec 15 2017, 1:01 AM

also:

T97593 "Fix beta cluster SSL certs issues" (private ticket but maybe it can be made public, dont know)

T50501 "beta: Get SSL certificates for *.{projects}.beta.wmflabs.org"

T75919 "Setup real ssl certs for Beta Cluster using a restricted project"

In the first ticket is a question whether it can be merged into the other 2 public ones.

Dzahn mentioned this in T75919: Setup real ssl certs for Beta Cluster using a restricted project.Dec 15 2017, 1:02 AM
Dzahn mentioned this in T50501: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org.
Krenair subscribed.Dec 15 2017, 7:51 AM

m.$project? I'm not sure those are useful domains? Which is probably why I
didn't put them on the list to get certs for

EddieGP added a comment.Dec 15 2017, 1:43 PM
In T182927#3839620, @Krenair wrote:

m.$project? I'm not sure those are useful domains?

Well they do the same thing they do in prod, redirecting to www.$project. zero.$project does the same btw and is probably the same level of useful that m. is. So if you want to mirror prod here, we should get a certificate for them. If we don't want those domains because they're useless in beta (which is a valid option), we should kill them altogether (remove them from DNS and Apache). Both is fine with me, but "resolving, redirecting to ssl and using a broken certificate" seems wrong.

Krenair added a comment.EditedDec 16 2017, 6:44 PM
In T182927#3840415, @EddieGP wrote:
In T182927#3839620, @Krenair wrote:

m.$project? I'm not sure those are useful domains?

Well they do the same thing they do in prod, redirecting to www.$project. zero.$project does the same btw and is probably the same level of useful that m. is. So if you want to mirror prod here, we should get a certificate for them. If we don't want those domains because they're useless in beta (which is a valid option), we should kill them altogether (remove them from DNS and Apache). Both is fine with me, but "resolving, redirecting to ssl and using a broken certificate" seems wrong.

Actually to mirror prod we'd need a wildcard certificate, which isn't available to us right now.

EddieGP added a comment.Dec 16 2017, 6:52 PM

Actually to mirror prod we'd need a wildcart certificate, which isn't available to us right now.

That wasn't my point. "To mirror prod" was meant to say "we want those domains to exist, because we also have them as subdomains under the real wikipedia.org/wiktionary.org/... domains". Basically my question is whether we should kill those domains completely or get an LE certificate for them.

Krenair added a comment.EditedDec 16 2017, 6:54 PM

Right now the plan is to leave everything as-is until it is possible to get a certificate that covers all subdomains (without anyone paying anything). I don't want to play whack-a-mole with missing redirect subdomains on the certs, and Let's Encrypt are expected to roll out support next month anyway.

Krenair claimed this task.Dec 16 2017, 6:58 PM
EddieGP added a comment.Dec 16 2017, 7:30 PM
In T182927#3842513, @Krenair wrote:

Right now the plan is to leave everything as-is until it is possible to get a certificate that covers all subdomains (without anyone paying anything). I don't want to play whack-a-mole with missing redirect subdomains on the certs, and Let's Encrypt are expected to roll out support next month anyway.

Thanks, with that context "don't do anything right now" is a lot easier to comprehend ;-)

Dzahn added a comment.Dec 18 2017, 5:25 PM

Yea, waiting for Letsencrypt wildcard certs to come seems the right way here. It should hopefully be soon.

Krenair added a comment.Dec 26 2017, 11:51 AM

https://letsencrypt.org/2017/12/07/looking-forward-to-2018.html

First, we’re planning to introduce an ACME v2 protocol API endpoint and support for wildcard certificates along with it. Wildcard certificates will be free and available globally just like our other certificates. We are planning to have a public test API endpoint up by January 4, and we’ve set a date for the full launch: Tuesday, February 27.

Krenair added a comment.Jan 4 2018, 11:55 PM
In T182927#3860911, @Krenair wrote:

https://letsencrypt.org/2017/12/07/looking-forward-to-2018.html

First, we’re planning to introduce an ACME v2 protocol API endpoint and support for wildcard certificates along with it. Wildcard certificates will be free and available globally just like our other certificates. We are planning to have a public test API endpoint up by January 4, and we’ve set a date for the full launch: Tuesday, February 27.

meh, nothing

Krenair added a comment.Jan 5 2018, 2:22 AM

https://community.letsencrypt.org/t/staging-endpoint-for-acme-v2/49605

Krenair merged a task: T173469: Non-existent wiki urls on beta cluster gives Unsafe/Insecure connection message.Jan 5 2018, 10:19 PM
Krenair added a subscriber: Niharika.
Krenair added a comment.EditedMar 13 2018, 11:08 PM

https://community.letsencrypt.org/t/acme-v2-and-wildcard-certificate-support-is-live/55579
We'll want to have a look at how much needs to change in acme_tiny to handle this (unless upstream is already there)

Edit: https://github.com/diafygi/acme-tiny/issues/195 - "@Cadair I don't plan on adding DNS challenge support (thus no wildcard support)" - will have to find something else then

gerritbot subscribed.Mar 24 2018, 4:32 PM

Change 421709 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] openstack: Permit deployment-prep-dns-manager to log in from instance subnet

https://gerrit.wikimedia.org/r/421709

gerritbot added a project: Patch-For-Review.Mar 24 2018, 4:32 PM
Krenair added a comment.EditedMar 24 2018, 4:37 PM

Once the commit above is approved, I've got a WIP script (in a local branch named acme-v2) based on a more modern (ACME v2 supporting) acme_tiny.py from https://github.com/diafygi/acme-tiny, into which I've mixed some of Adrien Dorsaz's DNS-01 changes from https://github.com/Trim/acme-dns-tiny/blob/master/acme_dns_tiny.py, along with some of my own to hook it up to the Designate API instead of TSIG keys.

bd808 subscribed.Mar 26 2018, 3:42 AM
gerritbot added a comment.Mar 26 2018, 2:24 PM

Change 421709 merged by Andrew Bogott:
[operations/puppet@production] openstack: Permit deployment-prep-dns-manager to log in from instance subnet

https://gerrit.wikimedia.org/r/421709

EddieGP renamed this task from various .beta.wmflabs.org domains use an invalid ssl certificate to Get letsencrypt wildcard cert for *.beta.wmflabs.org domains.Apr 10 2018, 5:51 PM
EddieGP updated the task description. (Show Details)
EddieGP mentioned this in T188288: fr.wikipedia.beta.wmflabs.org uses an invalid security certificate.Apr 10 2018, 5:55 PM
Krenair mentioned this in T194962: Create and deploy a centralized letsencrypt service.May 20 2018, 10:25 PM
Krenair closed subtask T184245: Create some mechanism for instances in projects to modify the project Designate records as Resolved.Jun 8 2018, 10:10 PM
Krenair added a comment.EditedJun 9 2018, 2:31 AM
krenair@deployment-cache-text04:~/acme-v2-test$ openssl x509 -in out.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            fa:cb:30:d6:02:4e:e7:49:20:2e:90:1f:f0:fe:29:bf:e0:be
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Fake LE Intermediate X1
        Validity
            Not Before: Jun  9 01:18:55 2018 GMT
            Not After : Sep  7 01:18:55 2018 GMT
        Subject: CN=beta.wmflabs.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a6:2c:3c:c4:10:3f:71:97:5d:1b:7c:d0:82:3d:
                    7f:ca:0a:09:c1:fa:f9:12:e6:1c:e4:f9:6b:d4:aa:
                    74:1f:a2:52:fb:a9:2c:f6:40:03:f9:15:c3:37:80:
                    ad:e0:20:90:05:04:f6:87:43:6f:7e:61:c5:85:ad:
                    52:ca:3a:a0:22:1e:fd:e5:9c:00:de:56:32:e7:54:
                    63:16:70:52:fe:6f:31:aa:82:69:22:fe:8d:9a:82:
                    a5:fa:8e:f9:23:ef:82:1f:8e:1b:a2:cd:7a:0d:62:
                    1d:52:3a:45:4a:f2:72:28:6c:8c:d9:ad:ee:cb:a4:
                    03:11:2a:45:88:91:f6:e9:00:7f:be:5c:8c:1d:15:
                    68:b6:aa:71:5f:06:7a:b5:9c:f5:89:a3:42:03:78:
                    67:d2:61:80:dd:b5:f2:7d:14:32:c1:1f:92:19:5b:
                    a9:0d:e1:88:fb:41:71:b5:09:71:64:9a:4c:7f:73:
                    77:ec:39:44:14:23:e1:50:69:32:b5:a8:e6:1f:d3:
                    c1:e1:17:8e:6b:ea:40:51:12:be:db:8d:91:04:5a:
                    6c:1a:76:e5:70:b3:bd:0d:b0:f5:51:1f:7f:45:af:
                    4e:cc:d8:21:32:41:81:69:20:8a:c9:23:ce:9e:3a:
                    01:97:d8:7d:b8:47:18:66:8f:37:81:d7:86:84:f5:
                    27:d7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                BE:1B:80:25:3F:50:EE:95:09:8B:89:77:1B:BD:C3:EB:31:3D:18:CF
            X509v3 Authority Key Identifier: 
                keyid:C0:CC:03:46:B9:58:20:CC:5C:72:70:F3:E1:2E:CB:20:A6:F5:68:3A

            Authority Information Access: 
                OCSP - URI:http://ocsp.stg-int-x1.letsencrypt.org
                CA Issuers - URI:http://cert.stg-int-x1.letsencrypt.org/

            X509v3 Subject Alternative Name: 
                DNS:*.m.wikibooks.beta.wmflabs.org, DNS:*.m.wikimedia.beta.wmflabs.org, DNS:*.m.wikinews.beta.wmflabs.org, DNS:*.m.wikipedia.beta.wmflabs.org, DNS:*.m.wikiquote.beta.wmflabs.org, DNS:*.m.wikisource.beta.wmflabs.org, DNS:*.m.wikiversity.beta.wmflabs.org, DNS:*.m.wikivoyage.beta.wmflabs.org, DNS:*.m.wiktionary.beta.wmflabs.org, DNS:*.wikibooks.beta.wmflabs.org, DNS:*.wikimedia.beta.wmflabs.org, DNS:*.wikinews.beta.wmflabs.org, DNS:*.wikipedia.beta.wmflabs.org, DNS:*.wikiquote.beta.wmflabs.org, DNS:*.wikisource.beta.wmflabs.org, DNS:*.wikiversity.beta.wmflabs.org, DNS:*.wikivoyage.beta.wmflabs.org, DNS:*.wiktionary.beta.wmflabs.org, DNS:*.zero.wikipedia.beta.wmflabs.org, DNS:beta.wmflabs.org, DNS:m.wikidata.beta.wmflabs.org, DNS:wikidata.beta.wmflabs.org
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org
                  User Notice:
                    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : DD:99:34:FC:A5:E7:24:80:C9:56:68:7D:81:34:99:08:
                                49:B2:49:F7:B5:69:D8:C7:BC:AB:3F:5C:C1:F3:6E:64
                    Timestamp : Jun  9 02:18:55.986 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:82:FA:3F:15:D7:DB:A9:3A:BF:FB:49:
                                D9:4A:03:82:8C:97:6F:BD:57:31:E4:13:BC:81:E0:49:
                                F9:2F:A4:26:D4:02:21:00:A5:DC:77:33:56:19:47:15:
                                0E:94:6C:5F:7F:7F:34:FE:2C:B4:DF:7F:6A:41:2A:98:
                                6A:5B:68:82:11:91:C1:99
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : B0:CC:83:E5:A5:F9:7D:6B:AF:7C:09:CC:28:49:04:87:
                                2A:C7:E8:8B:13:2C:63:50:B7:C6:FD:26:E1:6C:6C:77
                    Timestamp : Jun  9 02:18:56.064 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:22:45:08:32:38:1E:B5:6E:78:91:C5:A6:
                                67:F2:58:31:3B:ED:93:4A:47:14:60:BA:21:18:D1:04:
                                2A:4D:31:31:02:21:00:99:11:71:C0:AC:A4:44:F5:70:
                                9D:6E:6B:D1:E1:8E:F6:A3:53:84:41:0D:4C:08:25:B0:
                                C0:1A:9B:D5:08:63:7D
    Signature Algorithm: sha256WithRSAEncryption
         7d:53:73:f6:0c:68:0e:9f:e4:22:69:9a:a1:6a:f8:df:f9:2f:
         d9:21:c8:ab:4d:50:f3:62:3d:31:4e:bb:c8:84:c0:7a:ae:ba:
         26:92:8b:4c:ea:4d:b0:30:47:98:7c:7d:fc:5b:75:bd:40:83:
         34:c9:d3:d5:c0:b0:e0:74:b0:22:e4:2f:1e:bc:28:a9:38:4c:
         37:14:5b:00:7b:3e:ab:02:2a:b2:5b:12:6c:53:1e:bf:45:fa:
         33:85:82:8f:ee:92:1d:b6:e8:54:5e:14:b2:6a:33:19:f7:31:
         7e:c3:e6:79:52:56:57:c7:58:38:55:bc:e2:bc:3e:e6:8c:59:
         28:d3:a1:e8:a1:aa:a4:98:98:15:19:97:ff:53:0f:2b:0f:85:
         38:0d:6d:7e:a0:df:5c:a4:5d:9d:11:7e:fd:36:12:57:91:c4:
         26:2a:f0:fd:56:6d:97:71:9a:2e:9e:8f:60:fd:95:47:ed:72:
         58:d0:cc:f2:c6:42:6b:ed:fb:01:85:11:5d:df:64:1f:f9:e5:
         58:26:d0:a3:19:09:fb:05:d1:90:ac:c0:8f:21:74:fb:e9:11:
         ba:05:83:ab:40:d0:23:a9:09:34:e9:06:e5:f8:65:42:8e:87:
         66:28:58:10:ed:f0:94:ed:86:25:a0:5d:6d:f3:d2:c0:67:e8:
         57:eb:98:3c

This is essentially the result of:

cat <<EOT > csrcfg
[req]
distinguished_name=req_dn
req_extensions=SAN
prompt=no
[req_dn]
commonName=beta.wmflabs.org
[SAN]
subjectAltName=DNS:beta.wmflabs.org,DNS:*.wikimedia.beta.wmflabs.org,DNS:*.wikipedia.beta.wmflabs.org,DNS:*.wikibooks.beta.wmflabs.org,DNS:*.wiktionary.beta.wmflabs.org,DNS:*.wikinews.beta.wmflabs.org,DNS:*.wikiquote.beta.wmflabs.org,DNS:*.wikisource.beta.wmflabs.org,DNS:*.wikiversity.beta.wmflabs.org,DNS:*.wikivoyage.beta.wmflabs.org,DNS:*.m.wikimedia.beta.wmflabs.org,DNS:*.m.wikipedia.beta.wmflabs.org,DNS:*.m.wikibooks.beta.wmflabs.org,DNS:*.m.wiktionary.beta.wmflabs.org,DNS:*.m.wikinews.beta.wmflabs.org,DNS:*.m.wikiquote.beta.wmflabs.org,DNS:*.m.wikisource.beta.wmflabs.org,DNS:*.m.wikiversity.beta.wmflabs.org,DNS:*.m.wikivoyage.beta.wmflabs.org,DNS:*.zero.wikipedia.beta.wmflabs.org,DNS:wikidata.beta.wmflabs.org,DNS:m.wikidata.beta.wmflabs.org
EOT

sudo openssl req -new -sha256 -out csr -key /etc/acme/key/beta_wmflabs_org.key -config csrcfg

sudo python acme_tiny_openstackdns.py --account-key /etc/acme/acct/acct.key --csr csr --os_auth_url "http://labcontrol1001.wikimedia.org:5000/v3" --os_project deployment-prep --os_username deployment-prep-dns-manager --os_password redacted --os_zone_id 4482e09c-3d25-447f-b8e2-5aa2a105b60e --directory-url "https://acme-staging-v02.api.letsencrypt.org/directory"

Script currently takes a ridiculous amount of time to run because it does a *lot* of waiting for labs-ns0 and labs-ns1 to sync new recordsets. Can probably have it issue the create commands to designate and then check everything afterwards to speed it up dramatically (right now it does each one in turn, so has to wait a few minutes for each domain instead of waiting once for it to sync all the records).

Edit: Yeah that cut it down to around 30 seconds to a few minutes total. I've also run it against Let's Encrypt prod and got a theoretically publicly-trusted certificate. Might try installing it tomorrow (/later) for a bit.

Krenair mentioned this in T196797: When changes to a Designate zone occur (e.g. record creations/deletions), there is a brief period in which the entire zone is NXDOMAIN.Jun 9 2018, 3:22 AM
Dzahn awarded a token.Jun 9 2018, 6:00 AM
Stashbot subscribed.Jun 9 2018, 9:00 PM

Mentioned in SAL (#wikimedia-releng) [2018-06-09T21:00:59Z] <Krenair> Temporarily substituting certificates on deployment-cache-text04 for certs generated from T182927 to test

Krenair added a comment.Jun 9 2018, 9:02 PM
alex@alex-laptop:~$ openssl s_client -connect deployment.wikimedia.beta.wmflabs.org:443 | openssl x509 -text -noout
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = beta.wmflabs.org
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:8a:25:ef:04:c4:7f:7c:9a:9a:7a:d5:30:29:59:d2:03:2f
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Jun  9 02:17:33 2018 GMT
            Not After : Sep  7 02:17:33 2018 GMT
        Subject: CN=beta.wmflabs.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a6:2c:3c:c4:10:3f:71:97:5d:1b:7c:d0:82:3d:
                    7f:ca:0a:09:c1:fa:f9:12:e6:1c:e4:f9:6b:d4:aa:
                    74:1f:a2:52:fb:a9:2c:f6:40:03:f9:15:c3:37:80:
                    ad:e0:20:90:05:04:f6:87:43:6f:7e:61:c5:85:ad:
                    52:ca:3a:a0:22:1e:fd:e5:9c:00:de:56:32:e7:54:
                    63:16:70:52:fe:6f:31:aa:82:69:22:fe:8d:9a:82:
                    a5:fa:8e:f9:23:ef:82:1f:8e:1b:a2:cd:7a:0d:62:
                    1d:52:3a:45:4a:f2:72:28:6c:8c:d9:ad:ee:cb:a4:
                    03:11:2a:45:88:91:f6:e9:00:7f:be:5c:8c:1d:15:
                    68:b6:aa:71:5f:06:7a:b5:9c:f5:89:a3:42:03:78:
                    67:d2:61:80:dd:b5:f2:7d:14:32:c1:1f:92:19:5b:
                    a9:0d:e1:88:fb:41:71:b5:09:71:64:9a:4c:7f:73:
                    77:ec:39:44:14:23:e1:50:69:32:b5:a8:e6:1f:d3:
                    c1:e1:17:8e:6b:ea:40:51:12:be:db:8d:91:04:5a:
                    6c:1a:76:e5:70:b3:bd:0d:b0:f5:51:1f:7f:45:af:
                    4e:cc:d8:21:32:41:81:69:20:8a:c9:23:ce:9e:3a:
                    01:97:d8:7d:b8:47:18:66:8f:37:81:d7:86:84:f5:
                    27:d7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                BE:1B:80:25:3F:50:EE:95:09:8B:89:77:1B:BD:C3:EB:31:3D:18:CF
            X509v3 Authority Key Identifier: 
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name: 
                DNS:*.m.wikibooks.beta.wmflabs.org, DNS:*.m.wikimedia.beta.wmflabs.org, DNS:*.m.wikinews.beta.wmflabs.org, DNS:*.m.wikipedia.beta.wmflabs.org, DNS:*.m.wikiquote.beta.wmflabs.org, DNS:*.m.wikisource.beta.wmflabs.org, DNS:*.m.wikiversity.beta.wmflabs.org, DNS:*.m.wikivoyage.beta.wmflabs.org, DNS:*.m.wiktionary.beta.wmflabs.org, DNS:*.wikibooks.beta.wmflabs.org, DNS:*.wikimedia.beta.wmflabs.org, DNS:*.wikinews.beta.wmflabs.org, DNS:*.wikipedia.beta.wmflabs.org, DNS:*.wikiquote.beta.wmflabs.org, DNS:*.wikisource.beta.wmflabs.org, DNS:*.wikiversity.beta.wmflabs.org, DNS:*.wikivoyage.beta.wmflabs.org, DNS:*.wiktionary.beta.wmflabs.org, DNS:*.zero.wikipedia.beta.wmflabs.org, DNS:beta.wmflabs.org, DNS:m.wikidata.beta.wmflabs.org, DNS:wikidata.beta.wmflabs.org
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org
                  User Notice:
                    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : DB:74:AF:EE:CB:29:EC:B1:FE:CA:3E:71:6D:2C:E5:B9:
                                AA:BB:36:F7:84:71:83:C7:5D:9D:4F:37:B6:1F:BF:64
                    Timestamp : Jun  9 03:17:33.234 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:87:B3:1A:1E:42:76:E3:FE:45:DB:54:
                                69:E4:B1:2A:DD:B9:62:17:A6:5F:4C:33:19:C5:D9:C7:
                                DC:37:FA:A0:C8:02:21:00:F9:FE:A8:AC:49:7B:89:B6:
                                FC:55:4A:27:86:8D:82:DE:31:41:F7:59:2E:A2:7C:1F:
                                22:B1:45:97:9E:F8:FA:7A
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : 29:3C:51:96:54:C8:39:65:BA:AA:50:FC:58:07:D4:B7:
                                6F:BF:58:7A:29:72:DC:A4:C3:0C:F4:E5:45:47:F4:78
                    Timestamp : Jun  9 03:17:33.285 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:47:49:F9:88:EC:C2:F3:D1:1F:E9:E4:F5:
                                C2:89:9A:C9:ED:5B:9B:7D:8D:44:5F:54:E6:A2:65:56:
                                F4:E2:04:9F:02:20:36:41:9E:F6:4A:D8:98:3D:FF:79:
                                CE:FE:AC:B9:0E:9D:BC:70:73:FD:3A:A1:5A:73:7F:8A:
                                E6:8C:2B:FC:9F:41
    Signature Algorithm: sha256WithRSAEncryption
         8e:ef:bb:a3:06:65:25:ab:9c:f3:31:da:89:2b:35:25:d1:a3:
         81:eb:ef:6c:4c:46:7e:e4:38:97:d6:d1:4e:cf:0b:df:bb:c1:
         8d:88:54:02:c1:22:61:9a:61:38:0d:b3:b7:17:82:25:75:ed:
         b7:c9:5b:a6:8e:7c:bf:7d:cd:de:06:1b:6b:eb:5e:25:92:a3:
         59:99:dc:9a:8d:b2:e6:a3:7c:cd:52:4a:b8:e1:6c:26:8e:1a:
         5e:8d:bd:32:3d:d3:68:b7:a7:f2:99:c3:ea:17:86:3c:34:43:
         3d:51:56:68:a8:f5:1a:6d:03:de:52:6e:c6:ca:6a:4d:7d:4d:
         4b:ec:08:04:17:50:6a:33:25:f5:03:3a:4f:6f:50:ec:a8:ae:
         1a:fc:85:dc:3f:17:dd:78:3c:f5:41:66:f6:8a:46:ed:67:c7:
         e9:f1:51:ba:49:69:0c:8e:fc:b7:10:80:50:42:5e:61:96:11:
         c6:9a:07:e7:a6:50:91:67:e7:8a:b6:5f:ea:6a:68:7c:9d:51:
         1c:24:28:99:39:11:db:08:29:b9:fe:4e:6f:a8:81:d4:d6:c9:
         fd:df:51:9c:c0:0f:a0:83:a8:e1:1f:49:e2:c0:a7:61:7e:94:
         9a:ff:40:6b:db:a2:4c:66:84:7e:64:a0:44:67:6b:2d:ce:fb:
         99:da:8f:5d
^C
Krenair added a comment.EditedJun 9 2018, 9:18 PM

I've been comparing openssl s_client -connect meta.wikimedia.org:443 2>&1 | openssl x509 -text -noout | grep DNS: | sed -e 's/^ *//' | sed -r -e 's/DNS:([^,]*)(, )?/\1\n/g' with openssl s_client -connect deployment.wikimedia.beta.wmflabs.org:443 2>&1 | openssl x509 -text -noout | grep DNS: | sed -e 's/^ *//' | sed -r -e 's/DNS:([^,]*)(, )?/\1\n/g' and the things that prod covers that we don't need to are:

mediawiki.org
*.mediawiki.org
*.m.mediawiki.org
*.planet.wikimedia.org
wikimediafoundation.org
*.wikimediafoundation.org
*.m.wikimediafoundation.org
wmfusercontent.org
*.wmfusercontent.org
w.wiki

My one above covers m.wikidata.beta.wmflabs.org instead of *.wikidata.beta.wmflabs.org and *.m.wikidata.beta.wmflabs.org. This probably makes more sense as we don't have a direct equivalent of test.wikidata.org / test.m.wikidata.org.

My one above is covering beta.wmflabs.org which we have set up as a redirect, obviously prod can't do the equivalent (which would be hosting https://org. :))

So I think all we're missing is entries for:

wikipedia.beta.wmflabs.org
wikimedia.beta.wmflabs.org
wikibooks.beta.wmflabs.org
wikinews.beta.wmflabs.org
wikiquote.beta.wmflabs.org
wikisource.beta.wmflabs.org
wikiversity.beta.wmflabs.org
wikivoyage.beta.wmflabs.org
wiktionary.beta.wmflabs.org

Which I think would ideally host the portals in beta, if we could get our stuff in order and actually unify prod and beta apache config (but this is T1256).

Edit: And added those now too.

Krenair added a comment.EditedJul 8 2018, 7:46 PM

So the ordinary puppetised renewal ran overnight and removed my wildcard cert from usage. Guess I should try to find some way to puppetise this in a way that doesn't break everything else. Problem is there's some very OpenStack-specific customisation to acme_tiny etc. in there. This thing might be easier to do properly once the central LE service work (T194962) is done and DNS support (T194965) is in there with some generic backend system in place. But at least we have a proof of concept.

Krenair mentioned this in T134447: letsencrypt puppetization: upgrade for scalability.Jul 8 2018, 7:50 PM
Niharika unsubscribed.Jul 10 2018, 5:14 PM
Krenair mentioned this in T199387: Beta eswikibooks certificate issues.Jul 11 2018, 10:51 PM
Krenair added a subtask: T199387: Beta eswikibooks certificate issues.
MarcoAurelio changed the status of subtask T199387: Beta eswikibooks certificate issues from Open to Stalled.Jul 11 2018, 11:11 PM
Krinkle triaged this task as High priority.Jul 12 2018, 4:14 AM
Krinkle edited projects, added Release-Engineering-Team; removed Patch-For-Review.
Krinkle subscribed.

It seems Let's Encrypt now publicly supports wildcard domains. Are there known blockers to using those?

Krenair added a comment.Jul 12 2018, 7:39 AM

No I just need to puppetise my work to put them into use, see above

Krenair added a comment.Aug 22 2018, 6:02 PM

Actually the way forward here is probably certcentral

Krenair added a subtask: T206922: Write designate integration script for certcentral DNS challenges.Oct 13 2018, 5:06 PM
greg moved this task from INBOX to Watching / External on the Release-Engineering-Team board.Nov 21 2018, 12:35 AM
greg edited projects, added Release-Engineering-Team (Watching / External); removed Release-Engineering-Team.
Krenair added a comment.Mar 20 2019, 6:57 PM

P8247 Beta unified cert by acme-chief, March 2019
1root@deployment-acme-chief03:~# openssl x509 -in /var/lib/acme-chief/certs/unified/live/rsa-2048.crt -noout -text
2Certificate:
3 Data:
4 Version: 3 (0x2)
5 Serial Number:
6 03:cf:a5:c9:5d:72:51:fa:8a:ae:68:19:16:5e:e7:ec:05:1c
7 Signature Algorithm: sha256WithRSAEncryption
8 Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
9 Validity
10 Not Before: Mar 20 17:50:20 2019 GMT
11 Not After : Jun 18 17:50:20 2019 GMT
12 Subject: CN = *.wikimedia.beta.wmflabs.org
13 Subject Public Key Info:
14 Public Key Algorithm: rsaEncryption
15 RSA Public-Key: (2048 bit)
16 Modulus:
17 00:e8:b6:c7:08:8e:54:0e:3b:75:99:43:f7:e7:15:
18 27:83:e0:94:55:91:62:8b:f3:63:f9:8e:7f:f3:95:
19 77:c6:5f:44:dd:33:56:d6:96:2e:1e:bc:4b:99:0b:
20 5d:1f:84:04:c6:58:c8:9c:09:88:bd:69:92:97:66:
21 da:23:35:2e:be:6c:56:43:bc:18:c7:2b:0f:e4:a1:
22 54:b2:fd:9b:be:66:a9:4f:41:b8:ae:7d:8a:33:44:
23 4d:81:21:54:e1:15:de:f6:c2:00:4c:3d:e7:a2:0c:
24 3e:3a:f0:df:a0:36:04:96:24:8c:d0:8a:6f:12:7b:
25 62:15:2c:f3:60:c9:c2:08:80:3d:ff:1b:c0:31:bb:
26 b3:45:23:19:1a:f9:c8:88:83:38:b3:47:f8:15:6b:
27 9e:f9:9f:6b:f9:40:cd:4c:7b:46:b8:30:45:06:c1:
28 02:d2:00:71:ac:d4:d3:31:fe:c7:8b:59:f1:d6:af:
29 8d:ed:be:47:f1:15:b7:f0:83:80:9a:70:1c:dc:43:
30 af:76:8b:82:af:f3:49:ac:ba:8c:24:c4:df:03:51:
31 f6:6b:dc:d2:23:33:09:be:ab:7c:96:b8:7a:c0:fb:
32 ac:d4:87:23:2c:18:b0:f2:30:63:03:2b:26:96:1a:
33 b6:c7:fb:71:90:bd:2f:68:0e:e9:7e:aa:c7:1e:5f:
34 88:61
35 Exponent: 65537 (0x10001)
36 X509v3 extensions:
37 X509v3 Key Usage: critical
38 Digital Signature, Key Encipherment
39 X509v3 Extended Key Usage:
40 TLS Web Server Authentication, TLS Web Client Authentication
41 X509v3 Basic Constraints: critical
42 CA:FALSE
43 X509v3 Subject Key Identifier:
44 D4:4D:E0:D9:17:D4:64:A8:AC:A7:48:9D:4B:2E:88:17:09:64:C1:50
45 X509v3 Authority Key Identifier:
46 keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
47
48 Authority Information Access:
49 OCSP - URI:http://ocsp.int-x3.letsencrypt.org
50 CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
51
52 X509v3 Subject Alternative Name:
53 DNS:*.m.wikibooks.beta.wmflabs.org, DNS:*.m.wikimedia.beta.wmflabs.org, DNS:*.m.wikinews.beta.wmflabs.org, DNS:*.m.wikipedia.beta.wmflabs.org, DNS:*.m.wikiquote.beta.wmflabs.org, DNS:*.m.wikisource.beta.wmflabs.org, DNS:*.m.wikiversity.beta.wmflabs.org, DNS:*.m.wikivoyage.beta.wmflabs.org, DNS:*.m.wiktionary.beta.wmflabs.org, DNS:*.wikibooks.beta.wmflabs.org, DNS:*.wikimedia.beta.wmflabs.org, DNS:*.wikinews.beta.wmflabs.org, DNS:*.wikipedia.beta.wmflabs.org, DNS:*.wikiquote.beta.wmflabs.org, DNS:*.wikisource.beta.wmflabs.org, DNS:*.wikiversity.beta.wmflabs.org, DNS:*.wikivoyage.beta.wmflabs.org, DNS:*.wiktionary.beta.wmflabs.org, DNS:*.zero.wikipedia.beta.wmflabs.org, DNS:beta.wmflabs.org, DNS:m.wikidata.beta.wmflabs.org, DNS:wikidata.beta.wmflabs.org
54 X509v3 Certificate Policies:
55 Policy: 2.23.140.1.2.1
56 Policy: 1.3.6.1.4.1.44947.1.1.1
57 CPS: http://cps.letsencrypt.org
58
59 CT Precertificate SCTs:
60 Signed Certificate Timestamp:
61 Version : v1 (0x0)
62 Log ID : 74:7E:DA:83:31:AD:33:10:91:21:9C:CE:25:4F:42:70:
63 C2:BF:FD:5E:42:20:08:C6:37:35:79:E6:10:7B:CC:56
64 Timestamp : Mar 20 18:50:20.053 2019 GMT
65 Extensions: none
66 Signature : ecdsa-with-SHA256
67 30:45:02:21:00:D2:BC:24:07:39:EE:4C:A0:8C:97:40:
68 1C:64:DA:4D:CF:A7:32:E8:7E:75:7C:9C:6D:42:1C:1F:
69 63:1B:70:84:C1:02:20:13:BE:3B:F6:0B:F2:A0:4B:C3:
70 42:D7:48:8A:90:1B:EB:D9:32:68:FE:64:2A:C2:DD:E4:
71 23:77:21:C2:06:3D:2B
72 Signed Certificate Timestamp:
73 Version : v1 (0x0)
74 Log ID : 29:3C:51:96:54:C8:39:65:BA:AA:50:FC:58:07:D4:B7:
75 6F:BF:58:7A:29:72:DC:A4:C3:0C:F4:E5:45:47:F4:78
76 Timestamp : Mar 20 18:50:20.055 2019 GMT
77 Extensions: none
78 Signature : ecdsa-with-SHA256
79 30:45:02:20:53:85:45:43:FF:54:2E:47:6D:5A:81:EC:
80 75:EF:66:8E:AE:F8:0E:D6:BE:18:C8:74:FE:94:9D:29:
81 81:7E:25:C6:02:21:00:AE:A8:8F:C2:90:51:29:72:78:
82 8C:B7:3B:3D:8A:3B:1B:65:AB:1F:12:A4:B4:5A:D4:65:
83 96:E7:9E:DB:98:92:FD
84 Signature Algorithm: sha256WithRSAEncryption
85 8f:d5:95:7a:81:98:0c:66:a4:12:c8:3f:c8:24:b0:d6:87:78:
86 a8:47:1d:43:04:fa:28:1c:fb:cd:51:c8:09:a5:e3:60:b8:a9:
87 54:ee:c0:73:ed:16:ea:d9:b3:1a:ac:c9:12:4b:cd:52:13:4f:
88 9b:a4:08:39:9a:9e:a4:00:29:52:1c:8f:48:4e:73:f9:60:23:
89 36:f9:ba:28:8a:23:b8:20:10:31:85:d5:22:47:19:16:2d:fa:
90 7e:ba:0c:2e:5a:45:c4:29:91:81:a3:eb:0f:52:9f:d9:0f:25:
91 b2:f5:f0:d6:f8:07:79:f5:53:88:58:c5:b5:16:54:54:7b:8d:
92 58:6a:67:a4:7c:23:1f:e4:e3:23:4d:8d:33:1e:ee:5c:05:72:
93 30:c9:44:19:cc:e6:ea:21:f0:11:7c:65:57:3a:f2:99:c4:f4:
94 43:0c:00:6f:b8:13:d1:af:f2:05:a9:23:6b:aa:b5:dc:24:99:
95 4a:57:1d:49:1e:c1:75:dd:40:40:83:47:fd:f4:bb:fe:ed:ed:
96 9c:a9:b9:f8:25:86:4d:5c:88:da:c7:7e:e9:dd:5d:cc:f2:42:
97 bf:e3:32:85:68:17:02:a2:96:ed:3a:53:ff:8d:9b:96:d6:dd:
98 6b:2f:d4:48:a7:72:53:b1:ae:aa:a7:73:d4:76:ad:25:15:a4:
99 7a:87:04:c7

Krinkle unsubscribed.Mar 20 2019, 7:04 PM
Krenair added a comment.EditedMar 20 2019, 7:05 PM

(That said, in the process I think I may have stumbled upon a rather serious bug in the cloud DNS infrastructure that took beta down briefly until I ran the cleanup script for the validation records - it stopped serving the *.beta.wmflabs.org A record :|)
Edit: T218842: Creating a TXT record in Designate can break other records

gerritbot added a comment.Mar 21 2019, 2:40 AM

Change 497929 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Allow acme-chief to provide unified cert

https://gerrit.wikimedia.org/r/497929

gerritbot added a project: Patch-For-Review.Mar 21 2019, 2:40 AM
gerritbot added a comment.Apr 1 2019, 8:46 AM

Change 497929 merged by Vgutierrez:
[operations/puppet@production] Allow acme-chief to provide unified cert

https://gerrit.wikimedia.org/r/497929

Krenair added a comment.EditedApr 1 2019, 10:43 AM

So a remaining part of this - other than the patch for the subtask - is this puppet commit which makes our tlsproxy setup actually use the new cert:

commit c11873022ba2be05f88dcd9ddcdc5e05329729ae
Author: Alex Monk <krenair@gmail.com>
Date:   Mon Apr 1 09:10:06 2019 +0000

    [LOCAL HACK] USE ACME_CHIEF CERTS

diff --git a/modules/profile/manifests/cache/ssl/unified.pp b/modules/profile/manifests/cache/ssl/unified.pp
index 19fb5321bd..613aeff4af 100644
--- a/modules/profile/manifests/cache/ssl/unified.pp
+++ b/modules/profile/manifests/cache/ssl/unified.pp
@@ -28,12 +28,12 @@ class profile::cache::ssl::unified(
     } else {
         # TODO: generalize this a bit?
         $certs_active = [
-            "${ucv}-ecdsa-unified", "${ucv}-rsa-unified",
+#            "${ucv}-ecdsa-unified", "${ucv}-rsa-unified",
         ]
         # These certs are deployed to all caches and OCSP stapled,
         # ready for use in $certs_active as options
         $certs = [
-            'globalsign-2018-ecdsa-unified', 'globalsign-2018-rsa-unified',
+#            'globalsign-2018-ecdsa-unified', 'globalsign-2018-rsa-unified',
         ]
         tlsproxy::localssl { 'unified':
             server_name    => 'www.wikimedia.org',
gerritbot added a comment.Apr 1 2019, 10:53 PM

Change 500631 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Add hiera option to serve user traffic using acme-chief certs

https://gerrit.wikimedia.org/r/500631

Krenair added a comment.EditedApr 5 2019, 1:33 AM

I think we should try to get these commits merged and then call it done:

gerritbot added a comment.Apr 8 2019, 10:40 AM

Change 500631 merged by Vgutierrez:
[operations/puppet@production] profile::cache::ssl::unified: Allow passing certs/certs_active by hiera

https://gerrit.wikimedia.org/r/500631

gerritbot added a comment.Apr 17 2019, 2:54 PM

Change 504571 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] tlsproxy: Ensure OCSP stapling nginx reload hook present for acme-chief

https://gerrit.wikimedia.org/r/504571

Krenair mentioned this in T221268: Remove old letsencrypt puppet module.Apr 17 2019, 5:36 PM
gerritbot added a comment.Apr 18 2019, 8:50 AM

Change 504571 merged by Vgutierrez:
[operations/puppet@production] tlsproxy: Ensure OCSP stapling nginx reload hook present for acme-chief

https://gerrit.wikimedia.org/r/504571

Krenair closed subtask T206922: Write designate integration script for certcentral DNS challenges as Resolved.Apr 23 2019, 8:29 AM
Krenair closed this task as Resolved.Apr 23 2019, 8:36 AM
Krenair closed subtask T199387: Beta eswikibooks certificate issues as Resolved.
Krenair closed subtask T190781: Secure deployment-prep sudo access to prevent privilege escalation by dns-manager credentials as Invalid.Nov 25 2020, 1:25 AM
Krenair changed the status of subtask T190781: Secure deployment-prep sudo access to prevent privilege escalation by dns-manager credentials from Invalid to Declined.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits