Page MenuHomePhabricator

acme-chief crashes upon OCSP responder errors
Closed, ResolvedPublic

Description

May 15 13:06:56 acmechief1001 systemd[1]: Started acme-chief Service.
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: SIGHUP received
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: Missing/invalid DNS zone updater CMD timeout, using the default one: 60.00
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: Number of certificates per status: Counter({'VALID': 54, 'CERTIFICATE_STAGED': 2})
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: Starting main loop...
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: Refreshing live OCSP response for certificate non-canonical-redirect-3 / ec-prime256v1
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: Traceback (most recent call last):
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:   File "/usr/lib/python3/dist-packages/acme_chief/ocsp.py", line 71, in fetch_response
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:     http_request.raise_for_status()
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:     raise HTTPError(http_error_msg, response=self)
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: http://ocsp.int-x3.letsencrypt.org/
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: The above exception was the direct cause of the following exception:
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: Traceback (most recent call last):
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:   File "/usr/bin/acme-chief-backend", line 11, in <module>
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:     load_entry_point('acme-chief==0.24', 'console_scripts', 'acme-chief-backend')()
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:   File "/usr/lib/python3/dist-packages/acme_chief/acme_chief.py", line 908, in main
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:     ACMEChief().run()
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:   File "/usr/lib/python3/dist-packages/acme_chief/acme_chief.py", line 364, in run
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:     self.certificate_management()
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:   File "/usr/lib/python3/dist-packages/acme_chief/acme_chief.py", line 867, in certificate_management
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:     self._fetch_ocsp_response(cert_id, key_type_id)
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:   File "/usr/lib/python3/dist-packages/acme_chief/acme_chief.py", line 843, in _fetch_ocsp_response
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:     ocsp_response = ocsp_request.fetch_response()
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:   File "/usr/lib/python3/dist-packages/acme_chief/ocsp.py", line 73, in fetch_response
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]:     raise OCSPRequestError from request_error
May 15 13:06:57 acmechief1001 acme-chief-backend[28211]: acme_chief.ocsp.OCSPRequestError
May 15 13:06:57 acmechief1001 systemd[1]: acme-chief.service: Main process exited, code=exited, status=1/FAILURE
May 15 13:06:57 acmechief1001 systemd[1]: acme-chief.service: Failed with result 'exit-code'.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, May 15, 1:10 PM
Restricted Application added a project: Operations. · View Herald TranscriptFri, May 15, 1:10 PM
Vgutierrez moved this task from Triage to TLS on the Traffic board.Fri, May 15, 1:11 PM
Vgutierrez triaged this task as Medium priority.Fri, May 15, 1:45 PM

Change 596679 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] acme_chief: Handle OCSP Request issues

https://gerrit.wikimedia.org/r/596679

Change 596679 merged by Vgutierrez:
[operations/software/acme-chief@master] acme_chief: Handle OCSP Request issues

https://gerrit.wikimedia.org/r/596679

Change 596705 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] Release 0.25

https://gerrit.wikimedia.org/r/596705

Change 596705 merged by Vgutierrez:
[operations/software/acme-chief@master] Release 0.25

https://gerrit.wikimedia.org/r/596705

Change 596713 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] acme_chief: Handle OCSP Request issues

https://gerrit.wikimedia.org/r/596713

Change 596714 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] Release 0.25

https://gerrit.wikimedia.org/r/596714

Change 596715 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] debian: Add release 0.25 to the changelog

https://gerrit.wikimedia.org/r/596715

Change 596713 merged by Vgutierrez:
[operations/software/acme-chief@debian] acme_chief: Handle OCSP Request issues

https://gerrit.wikimedia.org/r/596713

Change 596714 merged by Vgutierrez:
[operations/software/acme-chief@debian] Release 0.25

https://gerrit.wikimedia.org/r/596714

Change 596715 merged by Vgutierrez:
[operations/software/acme-chief@debian] debian: Add release 0.25 to the changelog

https://gerrit.wikimedia.org/r/596715

Mentioned in SAL (#wikimedia-operations) [2020-05-15T17:55:44Z] <vgutierrez> upload acme-chief 0.25 to apt.wm.o (buster) - T252881

Vgutierrez closed this task as Resolved.Fri, May 15, 5:57 PM
Vgutierrez claimed this task.