Page MenuHomePhabricator

Toolforge: SSL support for new domain toolforge.org
Closed, ResolvedPublic

Description

In T234037: Toolforge ingress: decide on final layout of north-south proxy setup we decided that the way to go is to use dynamicproxy as front proxy for Toolforge. Thus, this proxy will need to handle SSL termination for both tools.wmflabs.org (currently using a wildcard cert) and for toolforge.org (also using a wildcard cert).

Ideally, we would start using acme-chief to generate these certs. This task is to clarify how this would be done and to perform the required steps.

Details

Related Gerrit Patches:

Event Timeline

aborrero created this task.Oct 11 2019, 8:56 AM

@Vgutierrez could you please advice on how to proceed with this?

aborrero triaged this task as Normal priority.Oct 11 2019, 8:57 AM
aborrero moved this task from Inbox to Important on the cloud-services-team (Kanban) board.

so, you should come with your own instances of acme-chief, the heavy work has been already done by @Krenair, he already implemented the acme-chief integration with the designate API to handle dns-01 challenges in the labs environment.

From a puppet point of view, the role handling acme-chief servers is https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/acme_chief/cloud.pp

For a working example you can check the acme-chief instances running on the deployment-prep project

ping @Krenair would you like to help me in this project? also T234617: Toolforge. introduce new domain toolforge.org

This is helpful for reference: T229786

Krenair added a comment.EditedOct 11 2019, 8:08 PM

ping @Krenair would you like to help me in this project? also T234617: Toolforge. introduce new domain toolforge.org

Yes. Before we decide anything at this level though the parent ticket, T234617: Toolforge. introduce new domain toolforge.org, says "Deciding who and how this domain will be hosted is another open question."
If it's going to be designate then we make a service user account for tools and give it permissions as was done in T229786: Create a service account to manage traffic.wmflabs.org. from acme-chief, then we make a couple of instances as set them up in a similar manner to deployment-acme-chief*

Actually the parent also links to https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/DNS_domain_usage#Resolution which says DNS will be managed via OpenStack Designate. for this domain

bd808 added a comment.Oct 11 2019, 8:31 PM

Actually the parent also links to https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/DNS_domain_usage#Resolution which says DNS will be managed via OpenStack Designate. for this domain

+1. I updated the parent task description to clarify this.

Krenair added a comment.EditedOct 11 2019, 11:00 PM

I've done some of the initial setup and begun thinking about what the rest of the process will be:

  • Created tools-acme-chief-0[12] as buster instances (acme-chief will not work on the version of python packaged in stretch or below) and done the usual dance with puppet to get them signed puppet certs.
  • Created a puppet prefix config in Horizon for tools-acme-chief:
authdns_servers:
- 208.80.154.135
- 208.80.154.11
http_proxy: ''
profile::acme_chief::accounts: {}
profile::acme_chief::active: tools-acme-chief-01.tools.eqiad.wmflabs
profile::acme_chief::passive: tools-acme-chief-02.tools.eqiad.wmflabs
profile::acme_chief::certificates: {}
shared_acme_certificates: {}
profile::acme_chief::challenges:
  dns-01:
    issuing_ca: letsencrypt.org
    ns_records:
    - cloud-ns0.wikimedia.org.
    - cloud-ns1.wikimedia.org.
    zone_update_cmd: /usr/local/bin/acme-chief-designate-sync.py
profile::acme_chief::cloud::designate_sync_auth_url: http://cloudcontrol1003.wikimedia.org:5000/v3
profile::acme_chief::cloud::designate_sync_project_name: tools
profile::acme_chief::cloud::designate_sync_region_name: eqiad1-r
profile::acme_chief::cloud::designate_sync_tidyup_enabled: true
profile::acme_chief::cloud::designate_sync_username: tools-dns-manager
  • Inserted tools-dns-manager password into puppet through a cherry-pick on tools-puppetmaster-01 adding profile::acme_chief::cloud::designate_sync_password to labs/private - done in commit 8fc478f85433ac68f8fbe72a25197adadf6a2102
  • TODO: If we're going to have the active-passive set up (I made the second host for it but am now wondering whether we think we need this outside prod/beta), generate the SSH public/private key pair for cert-sync, replace modules/secret/secrets/keyholder/authdns_acmechief and modules/secret/secrets/keyholder/authdns_acmechief.pub in tools-puppetmaster-01 cherry-picks. Any thoughts on whether we should bother with the passive node for tools @Vgutierrez?
  • TODO (done on -01 but not -02, pending above): Apply the role::acme_chief::cloud role on each of the instances individually (in my experience roles in prefix/project config can be problematic) and run puppet.
  • Ran the account creation code (I should probably get around to T207372: Add simple script for account creation) and insert into accounts dict in hiera and the private parts into a puppetmaster cherry-picks (Add the private key and regr JSON in a cherry-pick to modules/secret/secrets/acme_chief/accounts/{hash}/private_key.pem and modules/secret/secrets/acme_chief/accounts/{hash}/regr.json). The resulting account dict should look something like this:
{hash}:
  directory: https://acme-v02.api.letsencrypt.org/directory
  regr: '{"body": {}, "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/{number}"}'

Updated equivalent of the WIP account creation code used for generating this - note this talks about the eventual filesystem paths rather than Wikimedia's puppet paths:

from acme_chief import acme_requests
from cryptography.hazmat.primitives import serialization
acc = acme_requests.ACMEAccount.create('admin.acmechiefletsencrypt@tools.wmflabs.org', directory_url='https://acme-v02.api.letsencrypt.org/directory')
print('This goes in /etc/certcentral/config.yaml - account ID: {}'.format(acc.account_id))
print('This goes in /etc/certcentral/accounts/{}/regr.json: {}'.format(acc.account_id, acc.regr.json_dumps()))
print('This goes in /etc/certcentral/accounts/{}/private_key.pem:'.format(acc.account_id))
print(acc.key.key.private_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PrivateFormat.TraditionalOpenSSL,
    encryption_algorithm=serialization.NoEncryption(),
).decode('ascii'))

Created commit ee0ec37a843cdb89d3236c3a0f08da662e7868a6 containing account secrets.

toolforge:
  CN: toolforge.org
  SNI:
  - toolforge.org
  - '*.toolforge.org'
  authorized_regexes:
  - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$
  challenge: dns-01

Probably worth noting there's nothing limiting this to toolforge.org, it could be used to issue tools.wmflabs.org certs.

Is it possible for tools.wmflabs.org to be on the same cert as toolforge.org? I believe that was the ideal situation for our use case here, iirc.

Yep.

toolforge:
  CN: toolforge.org
  SNI:
  - toolforge.org
  - '*.toolforge.org'
  - tools.wmflabs.org
  - '*.tools.wmflabs.org'
  authorized_regexes:
  - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$
  challenge: dns-01

I've updated my comment above to reflect the current status, at some point I should turn it into proper docs for anyone else who needs to set acme-chief up with wikimedia puppetisation.
The remaining issues here are that:

  • toolforge.org. still has NS records in the org. zone pointing to prod - that's T235303: Update authoratiative nameservers for the toolforge.org domain to point to Designate - until this is done we can only get certs for tools.wmflabs.org.
  • we need to decide whether prod's setup of two nodes - active and passive, with a 'cert-sync' mechanism to keep passive up to date - makes sense here.
    • If it does then we should generate the cert-sync SSH key, cherry-pick that into labs/private on tools-puppetmaster, add the role::acme_chief::cloud role to tools-acme-chief-02 and run puppet there.
    • If not then we should introduce some flag to disable the sync mechanism to stop puppet on the active host from attempting to sync.

Yep.

toolforge:
  CN: toolforge.org
  SNI:
  - toolforge.org
  - '*.toolforge.org'
  - tools.wmflabs.org
  - '*.tools.wmflabs.org'
  authorized_regexes:
  - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$
  challenge: dns-01

Currently the toolforge front proxy is using *.wmflabs.org in the certificate. I believe the 2 you mentioned (tools.wmflabs.org and *.tools.wmflabs.org) are a good replacement, but we should double-check that we aren't leaving anything uncovered.
Here are some docs https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/SSL_certificates for additional context.

Specifically, the same current *.wmflabs.org cert is used also in tools-static* and in novaproxy-* servers (mind the cross-project usage).

Yep.

toolforge:
  CN: toolforge.org
  SNI:
  - toolforge.org
  - '*.toolforge.org'
  - tools.wmflabs.org
  - '*.tools.wmflabs.org'
  authorized_regexes:
  - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$
  challenge: dns-01

Currently the toolforge front proxy is using *.wmflabs.org in the certificate. I believe the 2 you mentioned (tools.wmflabs.org and *.tools.wmflabs.org) are a good replacement, but we should double-check that we aren't leaving anything uncovered.
Here are some docs https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/SSL_certificates for additional context.
Specifically, the same current *.wmflabs.org cert is used also in tools-static* and in novaproxy-* servers (mind the cross-project usage).

Yeeeeeeah. That's T206235.

Yep.

toolforge:
  CN: toolforge.org
  SNI:
  - toolforge.org
  - '*.toolforge.org'
  - tools.wmflabs.org
  - '*.tools.wmflabs.org'
  authorized_regexes:
  - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$
  challenge: dns-01

Currently the toolforge front proxy is using *.wmflabs.org in the certificate. I believe the 2 you mentioned (tools.wmflabs.org and *.tools.wmflabs.org) are a good replacement, but we should double-check that we aren't leaving anything uncovered.
Here are some docs https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/SSL_certificates for additional context.
Specifically, the same current *.wmflabs.org cert is used also in tools-static* and in novaproxy-* servers (mind the cross-project usage).

I dug around a bit, I found that sometimes paws requests end up at tools-proxy servers, not sure how:

root@tools-proxy-03:~# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | cut -f1 -d' ' | sort -d | uniq | grep wmflabs.org
instance-tools-proxy-03.tools.wmflabs.org
paws-beta.wmflabs.org
paws-public.wmflabs.org
tools.wmflabs.org
www.tools.wmflabs.org
root@tools-proxy-03:~#
root@tools-proxy-04:/var/log/nginx# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | cut -f1 -d' ' | sort -d | uniq | grep wmflabs.org
paws-public.wmflabs.org
root@tools-proxy-04:/var/log/nginx#

Meanwhile tools-static doesn't seem to use the same logging format, it doesn't include the host header. :/

Krenair added a comment.EditedOct 16 2019, 12:15 AM

Yesterday I got access to the paws project and went looking around. I found that the nginx config used on paws-proxy was responsible for proxying paws-beta/paws-public traffic through to tools-proxy. To set up that connection it just uses IP address and doesn't verify the TLS cert, so we shouldn't need to do anything special for it here.

For tools-static stuff:
<Krenair> next problem is going to be tools-static which does not appear to be logging host headers
<bd808> we can fix that in ops/puppet. The tools proxy didn't log vhost until I added it
<bd808> but nothing should be going there except tools-static.wmflabs.org
<bd808> which I think is handled via project-proxy?
<Krenair> nope looks like it's direct
<Krenair> tools-static.wmflabs.org has address 185.15.56.60
<Krenair> 60.56.15.185.in-addr.arpa domain name pointer instance-tools-static-13.tools.wmflabs.org.
<bd808> ah. ok. That's memory jogging. we have 2 instances and a hiera setting for which is active
<Krenair> if it's just a simple single host with no wildcard we can forget *.wmflabs.org and acme-chief, can just give it the script to do simple LE http-01
<Krenair> do wonder why it's not behind project-proxy though
<bd808> hysterical rasins?

Valentin, please could you address this and then assign it back to me?

  • we need to decide whether prod's setup of two nodes - active and passive, with a 'cert-sync' mechanism to keep passive up to date - makes sense here.
    • If it does then we should generate the cert-sync SSH key, cherry-pick that into labs/private on tools-puppetmaster, add the role::acme_chief::cloud role to tools-acme-chief-02 and run puppet there.
    • If not then we should introduce some flag to disable the sync mechanism to stop puppet on the active host from attempting to sync.
Krenair claimed this task.Mon, Oct 21, 8:55 PM

eh, I'll just do #1 for now. can always get rid of it later

Ran ssh-keygen -t ed25519 -f modules/secret/secrets/keyholder/authdns_acmechief on tools-puppetmaster (the file name gives away the other, original purpose of this key in prod :)) and put the passphrase in modules/secret/secrets/keyholder/authdns_acmechief.passphrase, committed the result, ran puppet on tools-acme-chief-01 and armed keyholder using the passphrase.
Added role::acme_chief::cloud role to tools-acme-chief-02 and running puppet.

Also armed keyholder on tools-acme-chief-02. Had to attempt to sudo sudo -u acme-chief ssh acme-chief@tools-acme-chief-02 on tools-acme-chief-01 to make it trust the SSH host key.

It turns out that the cert-sync script relies on the remote host having its SSH host key trusted, which is fine if your environment has PuppetDB (i.e. prod, deployment-prep), but tools does not (and cert-sync runs as the acme-chief user which doesn't have it's own user directory, so no user-specific known_hosts stuff either). Tools instead has this gem: https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/manifests/toolforge/grid/base.pp$61 - at first glance this only does the host it sits on, but $store is an NFS share. :)
I've applied that profile to both instances.

Had to usermod -a -G acme-chief acme-chief (followed by restarting the keyholder-proxy service) to get cert-sync auth with keyholder working - possibly puppet created the acme-chief user before our debian/postinst from the package did, so acme-chief was not a member of its own group? :/

Anyway puppet runs fine on both hosts so moving on to issuing a cert.

Set this in hiera:

profile::acme_chief::certificates:
toolforge:
  CN: toolforge.org
  SNI:
  - toolforge.org
  - '*.toolforge.org'
  - tools.wmflabs.org
  - '*.tools.wmflabs.org'
  authorized_regexes:
  - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$
  challenge: dns-01

Ran puppet on tools-acme-chief-01.

After sorting out a little error in the hiera I pasted above (neglected to change http_proxy which it turns out defaults to webproxy.eqiad.wmnet despite T122368), I've run into a new one:
Oct 21 22:34:53 tools-acme-chief-01 acme-chief-backend[29216]: Unexpected return code spawning DNS zone updater: 1
Oct 21 22:34:53 tools-acme-chief-01 acme-chief-backend[29216]: Failed to perform DNS zone update for certificate toolforge / rsa-2048

And when I try to run the command myself:

acme-chief@tools-acme-chief-01:/$ /usr/local/bin/acme-chief-designate-sync.py [etc. etc. etc.]
Traceback (most recent call last):
  File "/usr/local/bin/acme-chief-designate-sync.py", line 5, in <module>
    from designateclient.v2 import client as designateclient
ModuleNotFoundError: No module named 'designateclient'

Turns out, we only install python3-designateclient on stretch!

$ git grep python3-designateclient
modules/openstack/manifests/clientpackages/mitaka/stretch.pp:        'python3-designateclient',
modules/openstack/manifests/clientpackages/newton/stretch.pp:        'python3-designateclient',
modules/openstack/manifests/clientpackages/vms/newton/stretch.pp:        'python3-designateclient',

Change 545081 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] profile::acme_chief::cloud: Require python3-designateclient etc.

https://gerrit.wikimedia.org/r/545081

Krenair added a comment.EditedMon, Oct 21, 11:18 PM

I've cherry-picked the above on tools-puppetmaster so I can keep going. That said we could remove the cherry-pick now and it'd still work as the package is now installed.
I've realised I have to set authdns_servers (now updated in the hiera above) to the IPv4 addresses of cloud-ns*, because that nameserver has AAAA records for itself which we can't reach from within labs. (acme-chief attempts to query the nameservers directly to find out if they're serving our new record yet, before telling the ACME API we're ready for them to check)
So here's our cert:

# openssl x509 -in /var/lib/acme-chief/certs/toolforge/live/rsa-2048.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:e2:01:1e:96:91:fe:be:55:0a:97:21:53:d2:23:1a:c2:b9
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Oct 21 22:15:27 2019 GMT
            Not After : Jan 19 22:15:27 2020 GMT
        Subject: CN = toolforge.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:d8:d6:d2:08:fe:66:e3:01:67:47:4b:f1:03:4e:
                    83:d2:0a:ed:06:86:68:8f:27:67:35:be:72:76:9f:
                    e5:2b:46:5b:76:27:97:0d:d7:47:75:34:99:87:8b:
                    71:52:25:a0:cb:31:ae:47:da:6a:bc:9c:be:12:8e:
                    5b:14:07:b4:f3:ae:ac:9e:bc:68:bb:01:1e:78:a8:
                    7f:c6:10:99:ac:b3:21:aa:b1:f8:6d:27:ec:0c:68:
                    aa:c1:61:42:3f:74:62:d1:68:86:1c:72:e6:bc:a5:
                    ba:ff:02:ad:57:f3:e2:c5:55:19:b6:f2:bf:44:1e:
                    55:f0:ca:ef:b1:d0:0e:3c:88:9e:f5:6e:d7:c5:7a:
                    b5:86:64:e4:c2:99:a6:50:ff:02:a3:13:af:5f:39:
                    17:67:a5:09:43:a3:e6:69:65:0d:c6:51:70:fb:97:
                    08:88:40:ad:a2:0d:8c:68:87:87:20:f1:db:7c:d4:
                    7b:f9:dc:1a:a2:56:52:f4:93:41:59:e4:be:eb:2f:
                    d0:39:3f:d6:88:15:c8:fc:1f:58:17:50:96:ec:75:
                    a1:f0:f3:7e:fa:c2:c5:0b:c1:4a:0c:21:bf:4e:4e:
                    c5:fd:78:52:aa:f0:13:87:f9:ff:5f:45:9f:56:46:
                    a1:7b:8b:e4:eb:36:0b:3f:41:6c:79:66:47:cc:ca:
                    9f:6d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                72:C0:47:A4:B2:E2:03:B6:ED:BD:9B:E4:C4:BB:BC:88:8F:48:04:15
            X509v3 Authority Key Identifier: 
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name: 
                DNS:*.toolforge.org, DNS:*.tools.wmflabs.org, DNS:toolforge.org, DNS:tools.wmflabs.org
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 6F:53:76:AC:31:F0:31:19:D8:99:00:A4:51:15:FF:77:
                                15:1C:11:D9:02:C1:00:29:06:8D:B2:08:9A:37:D9:13
                    Timestamp : Oct 21 23:15:27.177 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:45:44:B8:F0:A4:47:7A:84:B6:53:11:92:
                                75:95:8C:6D:E9:32:06:BE:1B:BD:D8:42:69:CE:F0:96:
                                6C:85:90:69:02:21:00:8A:01:D3:B7:32:5D:E6:CE:55:
                                E8:42:50:00:65:35:FC:DE:B2:FA:EA:95:8C:89:0E:87:
                                A9:16:29:D6:3C:FB:2C
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 07:B7:5C:1B:E5:7D:68:FF:F1:B0:C6:1D:23:15:C7:BA:
                                E6:57:7C:57:94:B7:6A:EE:BC:61:3A:1A:69:D3:A2:1C
                    Timestamp : Oct 21 23:15:27.178 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:A9:E2:6A:F3:D2:A3:F5:C9:C4:EB:43:
                                7E:77:49:97:59:94:0C:E9:8F:44:1E:D3:40:1F:60:3E:
                                D8:3C:1D:F6:44:02:21:00:BD:71:8B:02:61:28:D2:E1:
                                A6:22:E3:31:B7:36:22:74:42:1F:61:BF:C5:46:04:E1:
                                B6:7D:7E:49:6C:25:86:A5
    Signature Algorithm: sha256WithRSAEncryption
         48:65:91:98:fb:94:82:8a:03:9a:10:e3:ce:a8:58:a5:40:67:
         99:83:4b:27:61:24:77:0e:ae:0c:cd:f5:5a:1c:5b:96:80:0a:
         3a:f4:19:b3:c9:21:7f:93:79:fb:17:16:58:94:95:a7:e8:a4:
         56:37:6d:17:34:02:c2:d6:2f:5e:f0:cc:0d:f9:0d:f0:a2:4c:
         e2:f5:3b:71:ea:20:02:cd:56:93:fa:41:e6:3e:7d:f5:dc:69:
         c6:54:75:b2:ca:1b:42:6b:7b:92:f2:6e:9a:52:3d:4c:5a:e1:
         b7:db:7f:75:a2:82:df:64:27:7c:e8:06:e1:37:19:1c:5e:81:
         58:0f:67:9e:7c:5d:82:b3:46:5a:60:6d:2c:d9:73:69:cd:d8:
         8d:62:b7:38:31:4d:6d:70:89:5d:1d:70:22:0e:75:5c:7c:58:
         66:46:1d:1e:89:3d:27:c4:2f:82:cc:b3:73:da:fa:a5:3f:22:
         d1:af:3d:5e:e8:9d:24:62:c8:41:d8:e7:bd:33:88:65:5d:28:
         93:d2:df:64:48:72:1f:52:20:90:62:80:c1:c4:23:11:a7:c3:
         f5:2d:bf:a6:22:d0:2a:d7:f7:2a:99:72:c4:78:d4:f0:f2:3e:
         34:98:5e:bf:d5:d2:b8:fa:e8:2f:c6:49:c2:48:0b:c9:53:b9:
         46:c0:e5:67

wow! awesome, thanks! I guess next step is to actually replace the cert and have this new one running.

Krenair added a comment.EditedTue, Oct 22, 7:20 PM

wow! awesome, thanks! I guess next step is to actually replace the cert and have this new one running.

Yeah, so the next step will be some puppet like acme_chief::cert { 'toolforge': } (probably also specifying puppet_svc) to have the proxy host pull down the cert and key from acme-chief, and then (for apache) some config like:

SSLCertificateFile /etc/acmecerts/toolforge/live/ec-prime256v1.crt
SSLCertificateChainFile /etc/acmecerts/toolforge/live/ec-prime256v1.chain.crt
SSLCertificateKeyFile /etc/acmecerts/toolforge/live/ec-prime256v1.key
SSLCertificateFile /etc/acmecerts/toolforge/live/rsa-2048.crt
SSLCertificateChainFile /etc/acmecerts/toolforge/live/rsa-2048.chain.crt
SSLCertificateKeyFile /etc/acmecerts/toolforge/live/rsa-2048.key

or for nginx maybe more like this:

ssl_certificate /etc/acmecerts/toolforge/live/rsa-2048.chained.crt;
ssl_certificate_key /etc/acmecerts/toolforge/live/rsa-2048.key;
ssl_certificate /etc/acmecerts/toolforge/live/ec-prime256v1.chained.crt;
ssl_certificate_key /etc/acmecerts/toolforge/live/ec-prime256v1.key;
ssl_stapling on;
ssl_stapling_file /etc/acmecerts/toolforge/live/rsa-2048.ocsp;
ssl_stapling_file /etc/acmecerts/toolforge/live/ec-prime256v1.ocsp;

etc.

Change 545679 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Swap toolforge proxies to use acme-chief certificates

https://gerrit.wikimedia.org/r/545679

bd808 moved this task from Triage to In Progress on the Toolforge board.Mon, Oct 28, 4:10 PM

Change 545081 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] profile::acme_chief::cloud: Require python3-designateclient etc.

https://gerrit.wikimedia.org/r/545081

Mentioned in SAL (#wikimedia-cloud) [2019-10-30T13:39:59Z] <arturo> icinga downtime toolschecker for 1h for replacing SSL cert T235252

Mentioned in SAL (#wikimedia-cloud) [2019-10-30T13:48:01Z] <arturo> replacing SSL cert in tools-proxy-x server (live-hacking https://gerrit.wikimedia.org/r/c/operations/puppet/+/545679 first for testing) T235252

Change 545679 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] Swap toolforge proxies to use acme-chief certificates

https://gerrit.wikimedia.org/r/545679

Mentioned in SAL (#wikimedia-cloud) [2019-10-30T13:53:22Z] <arturo> replacing SSL cert in tools-proxy-x server apparently OK (merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/545679) T235252

aborrero closed this task as Resolved.Wed, Oct 30, 1:55 PM

Apparently this is done! A big thanks to @Krenair for doing the hard work!

Krenair added a parent task: Restricted Task.Wed, Oct 30, 8:17 PM