In T234037: Toolforge ingress: decide on final layout of north-south proxy setup we decided that the way to go is to use dynamicproxy as front proxy for Toolforge. Thus, this proxy will need to handle SSL termination for both tools.wmflabs.org (currently using a wildcard cert) and for toolforge.org (also using a wildcard cert).
Ideally, we would start using acme-chief to generate these certs. This task is to clarify how this would be done and to perform the required steps.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | aborrero | T125589 Allow each tool to have its own subdomain for browser sandbox/cookie isolation | |||
| Resolved | aborrero | T234617 Toolforge. introduce new domain toolforge.org | |||
| Restricted Task | |||||
| Resolved | Krenair | T235252 Toolforge: SSL support for new domain toolforge.org | |||
| Resolved | None | T235304 Create a service account to manage toolforge.org. from acme-chief | |||
| Resolved | Andrew | T235303 Update authoratiative nameservers for the toolforge.org domain to point to Designate |
so, you should come with your own instances of acme-chief, the heavy work has been already done by @Krenair, he already implemented the acme-chief integration with the designate API to handle dns-01 challenges in the labs environment.
From a puppet point of view, the role handling acme-chief servers is https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/acme_chief/cloud.pp
For a working example you can check the acme-chief instances running on the deployment-prep project
ping @Krenair would you like to help me in this project? also T234617: Toolforge. introduce new domain toolforge.org
Yes. Before we decide anything at this level though the parent ticket, T234617: Toolforge. introduce new domain toolforge.org, says "Deciding who and how this domain will be hosted is another open question."
If it's going to be designate then we make a service user account for tools and give it permissions as was done in T229786: Create a service account to manage traffic.wmflabs.org. from acme-chief, then we make a couple of instances as set them up in a similar manner to deployment-acme-chief*
Actually the parent also links to https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/DNS_domain_usage#Resolution which says DNS will be managed via OpenStack Designate. for this domain
I've done some of the initial setup and begun thinking about what the rest of the process will be:
authdns_servers:
- 208.80.154.135
- 208.80.154.11
http_proxy: ''
profile::acme_chief::accounts: {}
profile::acme_chief::active: tools-acme-chief-01.tools.eqiad.wmflabs
profile::acme_chief::passive: tools-acme-chief-02.tools.eqiad.wmflabs
profile::acme_chief::certificates: {}
shared_acme_certificates: {}
profile::acme_chief::challenges:
dns-01:
issuing_ca: letsencrypt.org
ns_records:
- cloud-ns0.wikimedia.org.
- cloud-ns1.wikimedia.org.
zone_update_cmd: /usr/local/bin/acme-chief-designate-sync.py
profile::acme_chief::cloud::designate_sync_auth_url: http://cloudcontrol1003.wikimedia.org:5000/v3
profile::acme_chief::cloud::designate_sync_project_name: tools
profile::acme_chief::cloud::designate_sync_region_name: eqiad1-r
profile::acme_chief::cloud::designate_sync_tidyup_enabled: true
profile::acme_chief::cloud::designate_sync_username: tools-dns-manager{hash}:
directory: https://acme-v02.api.letsencrypt.org/directory
regr: '{"body": {}, "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/{number}"}'Updated equivalent of the WIP account creation code used for generating this - note this talks about the eventual filesystem paths rather than Wikimedia's puppet paths:
from acme_chief import acme_requests from cryptography.hazmat.primitives import serialization acc = acme_requests.ACMEAccount.create('admin.acmechiefletsencrypt@tools.wmflabs.org', directory_url='https://acme-v02.api.letsencrypt.org/directory') print('This goes in /etc/certcentral/config.yaml - account ID: {}'.format(acc.account_id)) print('This goes in /etc/certcentral/accounts/{}/regr.json: {}'.format(acc.account_id, acc.regr.json_dumps())) print('This goes in /etc/certcentral/accounts/{}/private_key.pem:'.format(acc.account_id)) print(acc.key.key.private_bytes( encoding=serialization.Encoding.PEM, format=serialization.PrivateFormat.TraditionalOpenSSL, encryption_algorithm=serialization.NoEncryption(), ).decode('ascii'))
Created commit ee0ec37a843cdb89d3236c3a0f08da662e7868a6 containing account secrets.
toolforge: CN: toolforge.org SNI: - toolforge.org - '*.toolforge.org' authorized_regexes: - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$ challenge: dns-01
Probably worth noting there's nothing limiting this to toolforge.org, it could be used to issue tools.wmflabs.org certs.
Is it possible for tools.wmflabs.org to be on the same cert as toolforge.org? I believe that was the ideal situation for our use case here, iirc.
Yep.
toolforge: CN: toolforge.org SNI: - toolforge.org - '*.toolforge.org' - tools.wmflabs.org - '*.tools.wmflabs.org' authorized_regexes: - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$ challenge: dns-01
I've updated my comment above to reflect the current status, at some point I should turn it into proper docs for anyone else who needs to set acme-chief up with wikimedia puppetisation.
The remaining issues here are that:
Currently the toolforge front proxy is using *.wmflabs.org in the certificate. I believe the 2 you mentioned (tools.wmflabs.org and *.tools.wmflabs.org) are a good replacement, but we should double-check that we aren't leaving anything uncovered.
Here are some docs https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/SSL_certificates for additional context.
Specifically, the same current *.wmflabs.org cert is used also in tools-static* and in novaproxy-* servers (mind the cross-project usage).
I dug around a bit, I found that sometimes paws requests end up at tools-proxy servers, not sure how:
root@tools-proxy-03:~# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | cut -f1 -d' ' | sort -d | uniq | grep wmflabs.org instance-tools-proxy-03.tools.wmflabs.org paws-beta.wmflabs.org paws-public.wmflabs.org tools.wmflabs.org www.tools.wmflabs.org root@tools-proxy-03:~#
root@tools-proxy-04:/var/log/nginx# cat /var/log/nginx/access.log /var/log/nginx/access.log.1 | cut -f1 -d' ' | sort -d | uniq | grep wmflabs.org paws-public.wmflabs.org root@tools-proxy-04:/var/log/nginx#
Meanwhile tools-static doesn't seem to use the same logging format, it doesn't include the host header. :/
Yesterday I got access to the paws project and went looking around. I found that the nginx config used on paws-proxy was responsible for proxying paws-beta/paws-public traffic through to tools-proxy. To set up that connection it just uses IP address and doesn't verify the TLS cert, so we shouldn't need to do anything special for it here.
For tools-static stuff:
<Krenair> next problem is going to be tools-static which does not appear to be logging host headers
<bd808> we can fix that in ops/puppet. The tools proxy didn't log vhost until I added it
<bd808> but nothing should be going there except tools-static.wmflabs.org
<bd808> which I think is handled via project-proxy?
<Krenair> nope looks like it's direct
<Krenair> tools-static.wmflabs.org has address 185.15.56.60
<Krenair> 60.56.15.185.in-addr.arpa domain name pointer instance-tools-static-13.tools.wmflabs.org.
<bd808> ah. ok. That's memory jogging. we have 2 instances and a hiera setting for which is active
<Krenair> if it's just a simple single host with no wildcard we can forget *.wmflabs.org and acme-chief, can just give it the script to do simple LE http-01
<Krenair> do wonder why it's not behind project-proxy though
<bd808> hysterical rasins?
Ran ssh-keygen -t ed25519 -f modules/secret/secrets/keyholder/authdns_acmechief on tools-puppetmaster (the file name gives away the other, original purpose of this key in prod :)) and put the passphrase in modules/secret/secrets/keyholder/authdns_acmechief.passphrase, committed the result, ran puppet on tools-acme-chief-01 and armed keyholder using the passphrase.
Added role::acme_chief::cloud role to tools-acme-chief-02 and running puppet.
Also armed keyholder on tools-acme-chief-02. Had to attempt to sudo sudo -u acme-chief ssh acme-chief@tools-acme-chief-02 on tools-acme-chief-01 to make it trust the SSH host key.
It turns out that the cert-sync script relies on the remote host having its SSH host key trusted, which is fine if your environment has PuppetDB (i.e. prod, deployment-prep), but tools does not (and cert-sync runs as the acme-chief user which doesn't have it's own user directory, so no user-specific known_hosts stuff either). Tools instead has this gem: https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/manifests/toolforge/grid/base.pp$61 - at first glance this only does the host it sits on, but $store is an NFS share. :)
I've applied that profile to both instances.
Had to usermod -a -G acme-chief acme-chief (followed by restarting the keyholder-proxy service) to get cert-sync auth with keyholder working - possibly puppet created the acme-chief user before our debian/postinst from the package did, so acme-chief was not a member of its own group? :/
Anyway puppet runs fine on both hosts so moving on to issuing a cert.
Set this in hiera:
profile::acme_chief::certificates: toolforge: CN: toolforge.org SNI: - toolforge.org - '*.toolforge.org' - tools.wmflabs.org - '*.tools.wmflabs.org' authorized_regexes: - ^tools-proxy-[0-9]+\.tools\.eqiad\.wmflabs$ challenge: dns-01
Ran puppet on tools-acme-chief-01.
After sorting out a little error in the hiera I pasted above (neglected to change http_proxy which it turns out defaults to webproxy.eqiad.wmnet despite T122368), I've run into a new one:
Oct 21 22:34:53 tools-acme-chief-01 acme-chief-backend[29216]: Unexpected return code spawning DNS zone updater: 1
Oct 21 22:34:53 tools-acme-chief-01 acme-chief-backend[29216]: Failed to perform DNS zone update for certificate toolforge / rsa-2048
And when I try to run the command myself:
acme-chief@tools-acme-chief-01:/$ /usr/local/bin/acme-chief-designate-sync.py [etc. etc. etc.]
Traceback (most recent call last):
File "/usr/local/bin/acme-chief-designate-sync.py", line 5, in <module>
from designateclient.v2 import client as designateclient
ModuleNotFoundError: No module named 'designateclient'Turns out, we only install python3-designateclient on stretch!
$ git grep python3-designateclient modules/openstack/manifests/clientpackages/mitaka/stretch.pp: 'python3-designateclient', modules/openstack/manifests/clientpackages/newton/stretch.pp: 'python3-designateclient', modules/openstack/manifests/clientpackages/vms/newton/stretch.pp: 'python3-designateclient',
Change 545081 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] profile::acme_chief::cloud: Require python3-designateclient etc.
I've cherry-picked the above on tools-puppetmaster so I can keep going. That said we could remove the cherry-pick now and it'd still work as the package is now installed.
I've realised I have to set authdns_servers (now updated in the hiera above) to the IPv4 addresses of cloud-ns*, because that nameserver has AAAA records for itself which we can't reach from within labs. (acme-chief attempts to query the nameservers directly to find out if they're serving our new record yet, before telling the ACME API we're ready for them to check)
So here's our cert:
# openssl x509 -in /var/lib/acme-chief/certs/toolforge/live/rsa-2048.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
03:e2:01:1e:96:91:fe:be:55:0a:97:21:53:d2:23:1a:c2:b9
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Validity
Not Before: Oct 21 22:15:27 2019 GMT
Not After : Jan 19 22:15:27 2020 GMT
Subject: CN = toolforge.org
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:d8:d6:d2:08:fe:66:e3:01:67:47:4b:f1:03:4e:
83:d2:0a:ed:06:86:68:8f:27:67:35:be:72:76:9f:
e5:2b:46:5b:76:27:97:0d:d7:47:75:34:99:87:8b:
71:52:25:a0:cb:31:ae:47:da:6a:bc:9c:be:12:8e:
5b:14:07:b4:f3:ae:ac:9e:bc:68:bb:01:1e:78:a8:
7f:c6:10:99:ac:b3:21:aa:b1:f8:6d:27:ec:0c:68:
aa:c1:61:42:3f:74:62:d1:68:86:1c:72:e6:bc:a5:
ba:ff:02:ad:57:f3:e2:c5:55:19:b6:f2:bf:44:1e:
55:f0:ca:ef:b1:d0:0e:3c:88:9e:f5:6e:d7:c5:7a:
b5:86:64:e4:c2:99:a6:50:ff:02:a3:13:af:5f:39:
17:67:a5:09:43:a3:e6:69:65:0d:c6:51:70:fb:97:
08:88:40:ad:a2:0d:8c:68:87:87:20:f1:db:7c:d4:
7b:f9:dc:1a:a2:56:52:f4:93:41:59:e4:be:eb:2f:
d0:39:3f:d6:88:15:c8:fc:1f:58:17:50:96:ec:75:
a1:f0:f3:7e:fa:c2:c5:0b:c1:4a:0c:21:bf:4e:4e:
c5:fd:78:52:aa:f0:13:87:f9:ff:5f:45:9f:56:46:
a1:7b:8b:e4:eb:36:0b:3f:41:6c:79:66:47:cc:ca:
9f:6d
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
72:C0:47:A4:B2:E2:03:B6:ED:BD:9B:E4:C4:BB:BC:88:8F:48:04:15
X509v3 Authority Key Identifier:
keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
Authority Information Access:
OCSP - URI:http://ocsp.int-x3.letsencrypt.org
CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
X509v3 Subject Alternative Name:
DNS:*.toolforge.org, DNS:*.tools.wmflabs.org, DNS:toolforge.org, DNS:tools.wmflabs.org
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 6F:53:76:AC:31:F0:31:19:D8:99:00:A4:51:15:FF:77:
15:1C:11:D9:02:C1:00:29:06:8D:B2:08:9A:37:D9:13
Timestamp : Oct 21 23:15:27.177 2019 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:20:45:44:B8:F0:A4:47:7A:84:B6:53:11:92:
75:95:8C:6D:E9:32:06:BE:1B:BD:D8:42:69:CE:F0:96:
6C:85:90:69:02:21:00:8A:01:D3:B7:32:5D:E6:CE:55:
E8:42:50:00:65:35:FC:DE:B2:FA:EA:95:8C:89:0E:87:
A9:16:29:D6:3C:FB:2C
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 07:B7:5C:1B:E5:7D:68:FF:F1:B0:C6:1D:23:15:C7:BA:
E6:57:7C:57:94:B7:6A:EE:BC:61:3A:1A:69:D3:A2:1C
Timestamp : Oct 21 23:15:27.178 2019 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:A9:E2:6A:F3:D2:A3:F5:C9:C4:EB:43:
7E:77:49:97:59:94:0C:E9:8F:44:1E:D3:40:1F:60:3E:
D8:3C:1D:F6:44:02:21:00:BD:71:8B:02:61:28:D2:E1:
A6:22:E3:31:B7:36:22:74:42:1F:61:BF:C5:46:04:E1:
B6:7D:7E:49:6C:25:86:A5
Signature Algorithm: sha256WithRSAEncryption
48:65:91:98:fb:94:82:8a:03:9a:10:e3:ce:a8:58:a5:40:67:
99:83:4b:27:61:24:77:0e:ae:0c:cd:f5:5a:1c:5b:96:80:0a:
3a:f4:19:b3:c9:21:7f:93:79:fb:17:16:58:94:95:a7:e8:a4:
56:37:6d:17:34:02:c2:d6:2f:5e:f0:cc:0d:f9:0d:f0:a2:4c:
e2:f5:3b:71:ea:20:02:cd:56:93:fa:41:e6:3e:7d:f5:dc:69:
c6:54:75:b2:ca:1b:42:6b:7b:92:f2:6e:9a:52:3d:4c:5a:e1:
b7:db:7f:75:a2:82:df:64:27:7c:e8:06:e1:37:19:1c:5e:81:
58:0f:67:9e:7c:5d:82:b3:46:5a:60:6d:2c:d9:73:69:cd:d8:
8d:62:b7:38:31:4d:6d:70:89:5d:1d:70:22:0e:75:5c:7c:58:
66:46:1d:1e:89:3d:27:c4:2f:82:cc:b3:73:da:fa:a5:3f:22:
d1:af:3d:5e:e8:9d:24:62:c8:41:d8:e7:bd:33:88:65:5d:28:
93:d2:df:64:48:72:1f:52:20:90:62:80:c1:c4:23:11:a7:c3:
f5:2d:bf:a6:22:d0:2a:d7:f7:2a:99:72:c4:78:d4:f0:f2:3e:
34:98:5e:bf:d5:d2:b8:fa:e8:2f:c6:49:c2:48:0b:c9:53:b9:
46:c0:e5:67wow! awesome, thanks! I guess next step is to actually replace the cert and have this new one running.
Yeah, so the next step will be some puppet like acme_chief::cert { 'toolforge': } (probably also specifying puppet_svc) to have the proxy host pull down the cert and key from acme-chief, and then (for apache) some config like:
SSLCertificateFile /etc/acmecerts/toolforge/live/ec-prime256v1.crt SSLCertificateChainFile /etc/acmecerts/toolforge/live/ec-prime256v1.chain.crt SSLCertificateKeyFile /etc/acmecerts/toolforge/live/ec-prime256v1.key SSLCertificateFile /etc/acmecerts/toolforge/live/rsa-2048.crt SSLCertificateChainFile /etc/acmecerts/toolforge/live/rsa-2048.chain.crt SSLCertificateKeyFile /etc/acmecerts/toolforge/live/rsa-2048.key
or for nginx maybe more like this:
ssl_certificate /etc/acmecerts/toolforge/live/rsa-2048.chained.crt; ssl_certificate_key /etc/acmecerts/toolforge/live/rsa-2048.key; ssl_certificate /etc/acmecerts/toolforge/live/ec-prime256v1.chained.crt; ssl_certificate_key /etc/acmecerts/toolforge/live/ec-prime256v1.key; ssl_stapling on; ssl_stapling_file /etc/acmecerts/toolforge/live/rsa-2048.ocsp; ssl_stapling_file /etc/acmecerts/toolforge/live/ec-prime256v1.ocsp;
etc.
Change 545679 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Swap toolforge proxies to use acme-chief certificates
Change 545081 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] profile::acme_chief::cloud: Require python3-designateclient etc.
Mentioned in SAL (#wikimedia-cloud) [2019-10-30T13:39:59Z] <arturo> icinga downtime toolschecker for 1h for replacing SSL cert T235252
Mentioned in SAL (#wikimedia-cloud) [2019-10-30T13:48:01Z] <arturo> replacing SSL cert in tools-proxy-x server (live-hacking https://gerrit.wikimedia.org/r/c/operations/puppet/+/545679 first for testing) T235252
Change 545679 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] Swap toolforge proxies to use acme-chief certificates
Mentioned in SAL (#wikimedia-cloud) [2019-10-30T13:53:22Z] <arturo> replacing SSL cert in tools-proxy-x server apparently OK (merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/545679) T235252
Yay. https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/547293/ to clean up the old absenting resource.
Change 572665 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] icinga: remove wmflabs.org HTTPS cert check
Change 572665 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] icinga: remove wmflabs.org HTTPS cert check