Page MenuHomePhabricator

Implement server-side OCSP stapling
Open, NormalPublic

Description

Provide centralized OCSP stapling as part of acme-chief backend duties

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 1 2019, 9:29 AM

Change 500397 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] tlsproxy: Allow update-ocsp-all writing in /etc/acmecerts

https://gerrit.wikimedia.org/r/500397

Change 500397 merged by Vgutierrez:
[operations/puppet@production] tlsproxy: Allow update-ocsp-all writing in /etc/acmecerts

https://gerrit.wikimedia.org/r/500397

Vgutierrez triaged this task as Normal priority.Apr 1 2019, 10:56 AM
Vgutierrez updated the task description. (Show Details)

Change 516604 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] x509: Expose the OCSP URI of a Certificate as a property

https://gerrit.wikimedia.org/r/516604

Change 529202 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] ocsp: Provide basic functionality to perform OCSP requests

https://gerrit.wikimedia.org/r/529202

Change 516604 merged by jenkins-bot:
[operations/software/acme-chief@master] x509: Expose the OCSP URI of a Certificate as a property

https://gerrit.wikimedia.org/r/516604

Change 529202 merged by jenkins-bot:
[operations/software/acme-chief@master] ocsp: Provide basic functionality to perform OCSP requests

https://gerrit.wikimedia.org/r/529202

Change 530464 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] ocsp: Allow to load an existing OCSPResponse from disk

https://gerrit.wikimedia.org/r/530464

Change 530465 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] acme_chief: Provide OCSP responses

https://gerrit.wikimedia.org/r/530465

Change 530548 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] ocsp: Provide basic test coverage

https://gerrit.wikimedia.org/r/530548

Change 530806 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] api: Allow acme-chief clients to fetch OCSP responses

https://gerrit.wikimedia.org/r/530806

Change 530464 merged by jenkins-bot:
[operations/software/acme-chief@master] ocsp: Allow to load an existing OCSPResponse from disk

https://gerrit.wikimedia.org/r/530464

Change 530548 merged by jenkins-bot:
[operations/software/acme-chief@master] ocsp: Provide basic test coverage

https://gerrit.wikimedia.org/r/530548

Change 530465 merged by jenkins-bot:
[operations/software/acme-chief@master] acme_chief: Provide OCSP responses

https://gerrit.wikimedia.org/r/530465

Change 530806 merged by jenkins-bot:
[operations/software/acme-chief@master] api: Allow acme-chief clients to fetch OCSP responses

https://gerrit.wikimedia.org/r/530806

Change 533856 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] Release 0.21

https://gerrit.wikimedia.org/r/533856

Change 533856 merged by Vgutierrez:
[operations/software/acme-chief@master] Release 0.21

https://gerrit.wikimedia.org/r/533856

Change 536006 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] x509: Expose the OCSP URI of a Certificate as a property

https://gerrit.wikimedia.org/r/536006

Change 536007 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] ocsp: Provide basic functionality to perform OCSP requests

https://gerrit.wikimedia.org/r/536007

Change 536008 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] ocsp: Allow to load an existing OCSPResponse from disk

https://gerrit.wikimedia.org/r/536008

Change 536009 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] ocsp: Provide basic test coverage

https://gerrit.wikimedia.org/r/536009

Change 536010 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] acme_chief: Provide OCSP responses

https://gerrit.wikimedia.org/r/536010

Change 536011 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] api: Allow acme-chief clients to fetch OCSP responses

https://gerrit.wikimedia.org/r/536011

Change 536012 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] Release 0.21

https://gerrit.wikimedia.org/r/536012

Change 536006 merged by jenkins-bot:
[operations/software/acme-chief@debian] x509: Expose the OCSP URI of a Certificate as a property

https://gerrit.wikimedia.org/r/536006

Change 536007 merged by jenkins-bot:
[operations/software/acme-chief@debian] ocsp: Provide basic functionality to perform OCSP requests

https://gerrit.wikimedia.org/r/536007

Change 536008 merged by jenkins-bot:
[operations/software/acme-chief@debian] ocsp: Allow to load an existing OCSPResponse from disk

https://gerrit.wikimedia.org/r/536008

Change 536009 merged by jenkins-bot:
[operations/software/acme-chief@debian] ocsp: Provide basic test coverage

https://gerrit.wikimedia.org/r/536009

Change 536010 merged by jenkins-bot:
[operations/software/acme-chief@debian] acme_chief: Provide OCSP responses

https://gerrit.wikimedia.org/r/536010

Change 536011 merged by jenkins-bot:
[operations/software/acme-chief@debian] api: Allow acme-chief clients to fetch OCSP responses

https://gerrit.wikimedia.org/r/536011

Change 536012 merged by jenkins-bot:
[operations/software/acme-chief@debian] Release 0.21

https://gerrit.wikimedia.org/r/536012

Change 536015 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] debian: Add release 0.21 to changelog

https://gerrit.wikimedia.org/r/536015

Change 536015 merged by Vgutierrez:
[operations/software/acme-chief@debian] debian: Add release 0.21 to changelog

https://gerrit.wikimedia.org/r/536015

Mentioned in SAL (#wikimedia-operations) [2019-09-12T07:45:36Z] <vgutierrez> uploaded acme-chief 0.21 to apt.wikimedia.org (buster) - T219765

Mentioned in SAL (#wikimedia-operations) [2019-09-12T08:22:54Z] <vgutierrez> upgrading to acme-chief 0.21 on acmechief-test instances - T219765

After upgrading acme-chief on acmechief-test1001, a tiny storm of OCSP requests was generated:

Sep 12 08:23:24 acmechief-test1001 acme-chief-backend[457]: Missing/invalid DNS zone updater CMD timeout, using the default one: 60.00
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: Number of certificates per status: Counter({'VALID': 48})
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: Starting main loop...
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate apt / ec-prime256v1
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for apt / ec-prime256v1
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate apt / rsa-2048
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for apt / rsa-2048
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate archiva / ec-prime256v1
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for archiva / ec-prime256v1
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate archiva / rsa-2048
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for archiva / rsa-2048
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate cloudelastic / ec-prime256v1
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for cloudelastic / ec-prime256v1
Sep 12 08:23:25 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate cloudelastic / rsa-2048
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for cloudelastic / rsa-2048
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate dumps / ec-prime256v1
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for dumps / ec-prime256v1
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate dumps / rsa-2048
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for dumps / rsa-2048
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate gerrit / ec-prime256v1
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for gerrit / ec-prime256v1
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate gerrit / rsa-2048
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for gerrit / rsa-2048
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate icinga / ec-prime256v1
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for icinga / ec-prime256v1
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate icinga / rsa-2048
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for icinga / rsa-2048
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate idp / ec-prime256v1
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for idp / ec-prime256v1
Sep 12 08:23:26 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate idp / rsa-2048
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for idp / rsa-2048
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate ldap / ec-prime256v1
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for ldap / ec-prime256v1
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate ldap / rsa-2048
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for ldap / rsa-2048
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate ldap-codfw1dev / ec-prime256v1
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for ldap-codfw1dev / ec-prime256v1
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate ldap-codfw1dev / rsa-2048
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for ldap-codfw1dev / rsa-2048
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate ldap-labtest / ec-prime256v1
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for ldap-labtest / ec-prime256v1
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate ldap-labtest / rsa-2048
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for ldap-labtest / rsa-2048
Sep 12 08:23:27 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate librenms / ec-prime256v1
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for librenms / ec-prime256v1
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate librenms / rsa-2048
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for librenms / rsa-2048
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate lists / ec-prime256v1
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for lists / ec-prime256v1
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate lists / rsa-2048
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for lists / rsa-2048
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate mirrors / ec-prime256v1
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for mirrors / ec-prime256v1
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate mirrors / rsa-2048
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for mirrors / rsa-2048
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate mx / ec-prime256v1
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for mx / ec-prime256v1
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate mx / rsa-2048
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for mx / rsa-2048
Sep 12 08:23:28 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate netbox / ec-prime256v1
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for netbox / ec-prime256v1
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate netbox / rsa-2048
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for netbox / rsa-2048
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-1 / ec-prime256v1
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-1 / ec-prime256v1
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-1 / rsa-2048
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-1 / rsa-2048
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-2 / ec-prime256v1
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-2 / ec-prime256v1
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-2 / rsa-2048
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-2 / rsa-2048
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-3 / ec-prime256v1
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-3 / ec-prime256v1
Sep 12 08:23:29 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-3 / rsa-2048
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-3 / rsa-2048
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-4 / ec-prime256v1
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-4 / ec-prime256v1
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-4 / rsa-2048
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-4 / rsa-2048
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-5 / ec-prime256v1
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-5 / ec-prime256v1
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-5 / rsa-2048
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-5 / rsa-2048
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-6 / ec-prime256v1
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-6 / ec-prime256v1
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate non-canonical-redirect-6 / rsa-2048
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for non-canonical-redirect-6 / rsa-2048
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate tendril / ec-prime256v1
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for tendril / ec-prime256v1
Sep 12 08:23:30 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate tendril / rsa-2048
Sep 12 08:23:31 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for tendril / rsa-2048
Sep 12 08:23:31 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate unified / ec-prime256v1
Sep 12 08:23:31 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for unified / ec-prime256v1
Sep 12 08:23:31 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate unified / rsa-2048
Sep 12 08:23:31 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for unified / rsa-2048
Sep 12 08:23:31 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate wikibase / ec-prime256v1
Sep 12 08:23:31 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for wikibase / ec-prime256v1
Sep 12 08:23:31 acmechief-test1001 acme-chief-backend[457]: Refreshing live OCSP response for certificate wikibase / rsa-2048
Sep 12 08:23:31 acmechief-test1001 acme-chief-backend[457]: live OCSP response refreshed successfully for wikibase / rsa-2048

and everything looks good:

root@acmechief-test1001:~# openssl ocsp -respin /var/lib/acme-chief/certs/unified/live/rsa-2048.ocsp -issuer /var/lib/acme-chief/certs/unified/live/rsa-2048.chain.crt -text
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: CN = Fake LE Intermediate X1
    Produced At: Sep 11 15:00:00 2019 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: C29C130A07D1FF36475F8766B701C13205DF6527
      Issuer Key Hash: C0CC0346B95820CC5C7270F3E12ECB20A6F5683A
      Serial Number: FA7F94D7577501D47CFCA058E7933A341C88
    Cert Status: good
    This Update: Sep 11 15:00:00 2019 GMT
    Next Update: Sep 18 15:00:00 2019 GMT

    Signature Algorithm: sha256WithRSAEncryption
         05:dc:94:ca:a5:21:18:4d:8a:e0:93:2d:8c:65:13:10:43:2a:
         85:58:df:f6:1d:7a:15:a7:f6:de:d0:f3:f7:45:12:7a:26:67:
         eb:bb:73:ee:e2:1b:3b:86:f5:49:1d:c3:84:a0:34:82:a9:ba:
         81:e4:4f:20:ba:a4:35:67:90:f1:ac:13:b4:b2:a6:6e:41:bf:
         1e:33:ca:e1:eb:f9:d4:9b:3e:2e:2e:29:65:ca:8d:56:b7:ac:
         2c:81:5b:15:a3:29:2b:5d:19:33:70:23:9b:f7:c7:dc:73:ac:
         73:eb:22:09:d6:8b:1c:6d:96:7f:65:ea:fe:12:bd:d0:2b:ba:
         2d:77:12:1a:8e:d1:63:fe:80:7a:ec:3d:0f:d1:c2:84:d2:c5:
         7a:b7:4c:bf:91:01:3c:4d:2f:c8:2b:d8:7d:c9:99:1e:ac:76:
         8f:c9:21:eb:77:75:cd:a1:4c:4f:8b:53:95:35:5d:06:57:b6:
         60:e4:d6:8f:e3:ae:ce:27:0d:c3:de:41:a1:72:5d:29:fc:bc:
         6b:e0:f7:d4:03:c8:53:19:85:90:a6:2c:aa:62:c5:e8:c1:2b:
         6a:19:7a:df:02:d1:f9:ac:71:dc:7c:4c:27:0d:e1:21:62:70:
         46:17:3a:8f:5a:97:46:a1:96:4f:8d:f4:24:af:ac:54:84:46:
         87:aa:4a:26
Response verify OK

Mentioned in SAL (#wikimedia-operations) [2019-09-16T10:16:36Z] <vgutierrez> upgrade acme-chief production servers to acme-chief 0.21 - T219765