Page MenuHomePhabricator

The TLS certificate for https://wma.wmflabs.org is expired
Closed, ResolvedPublic

Description

The TLS certificate for https://wma.wmflabs.org/ expired on 9/7/2020, 6:03:09 PM (Eastern Daylight Time). This tool is linked from any Wikipedia article with coordinates, making it highly visible. It's been expired for 5 hours and one complaint has already made it to IRC.

Event Timeline

aborrero triaged this task as High priority.Sep 8 2020, 8:55 AM
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Mentioned in SAL (#wikimedia-cloud) [2020-09-08T09:10:34Z] <arturo> restart acme-chief service in roject-proxy-acme-chief-01 (T262237)

Mentioned in SAL (#wikimedia-cloud) [2020-09-08T09:18:07Z] <arturo> upgrading acme-chief deb package from 0.25-1 to 0.28-1 on project-proxy-acme-chief-01 (T262237)

aborrero added a subscriber: aborrero.

The acme-chief backend gets OCSPResponseStatus.UNAUTHORIZED and can't generate the new certs apparently.

Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]: Waiting till tiles / ec-prime256v1 is generated to be able to push the new certificate
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]: Traceback (most recent call last):
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:   File "/usr/bin/acme-chief-backend", line 11, in <module>
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:     load_entry_point('acme-chief==0.28', 'console_scripts', 'acme-chief-backend')()
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:   File "/usr/lib/python3/dist-packages/acme_chief/acme_chief.py", line 928, in main
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:     ACMEChief().run()
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:   File "/usr/lib/python3/dist-packages/acme_chief/acme_chief.py", line 377, in run
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:     self.certificate_management()
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:   File "/usr/lib/python3/dist-packages/acme_chief/acme_chief.py", line 906, in certificate_management
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:     self._fetch_ocsp_response(cert_id, key_type_id)
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:   File "/usr/lib/python3/dist-packages/acme_chief/acme_chief.py", line 849, in _fetch_ocsp_response
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:     if ocsp_response.next_update - datetime.datetime.utcnow() < cert_details['ocsp_update_threshold']:
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:   File "/usr/lib/python3/dist-packages/acme_chief/ocsp.py", line 105, in next_update
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:     return self._response.next_update
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:   File "/usr/lib/python3/dist-packages/cryptography/hazmat/backends/openssl/ocsp.py", line 30, in wrapper
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]:     "OCSP response status is not successful so the property "
Sep 08 09:29:24 project-proxy-acme-chief-01 acme-chief-backend[15987]: ValueError: OCSP response status is not successful so the property has no value

But this is for the 'tiles' certificate set, and not for wma? weird.

Mentioned in SAL (#wikimedia-cloud) [2020-09-08T10:05:52Z] <arturo> remove /var/lib/acme-chief/certs/* to force acme-chief generating new certs instead of renewing them (T262237)

aborrero added a subscriber: Vgutierrez.

This should be fixed now. Thanks @Vgutierrez for the assistance on IRC.

image.png (433×1 px, 135 KB)

Closing task now, thanks @AntiCompositeNumber feel free to reopen if required.