[epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy
Closed, ResolvedPublic
Actions

Description

Tracking task.

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		bd808	T273956 acme-chief sometimes doesn't refresh certificates because it ignores SIGHUP
Resolved		Vgutierrez	T293585 [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy
Resolved	BUG REPORT	Urbanecm	T293070 upload.wikimedia.beta.wmflabs.org certificate expired (October 2021)
Resolved		Vgutierrez	T271808 The certificate for upload.beta.wmflabs.org expired on January 12, 2021.
Resolved		Krenair	T267858 The certificate for upload.beta.wmflabs.org expired on November 13, 2020.
Duplicate	BUG REPORT	None	T293251 The certificate for upload.wikimedia.beta.wmflabs.org expired on October 9, 2021.
Resolved		None	T262816 The certificate for en.wikipedia.beta.wmflabs.org expired on 2020-09-14
Resolved	BUG REPORT	Krenair	T257968 Certificate for *.beta.wmflabs.org has expired (July 2020)
Resolved		Vgutierrez	T259338 do not generate metadata for parts that aren't allowed
Duplicate		None	T262806 Beta cluster certificates have expired (September 2020)
Resolved	BUG REPORT	None	T296000 *.beta.wmflabs.org Certificate has expired (November 2021 edition)
Resolved	BUG REPORT	MatthewVernon	T301995 The certificate for upload.wikimedia.beta.wmflabs.org expired on February 16, 2022.
Resolved	BUG REPORT	None	T306492 *.beta.wmflabs.org Certificate has expired (April 2022 edition)
Resolved		ori	T310957 *.beta.wmflabs.org Certificate has expired (June 2022 edition)
Resolved	BUG REPORT	AlexisJazz	T337642 upload.wikimedia.beta.wmflabs.org certificate expired (May 2023)

Event Timeline

AlexisJazz created this task.Oct 17 2021, 9:55 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 17 2021, 9:55 PM

AlexisJazz added subtasks: T293070: upload.wikimedia.beta.wmflabs.org certificate expired (October 2021), T271808: The certificate for upload.beta.wmflabs.org expired on January 12, 2021., T267858: The certificate for upload.beta.wmflabs.org expired on November 13, 2020..Oct 17 2021, 9:56 PM

AlexisJazz renamed this task from [epic] The certificate for upload.wikimedia.beta.wmflabs.org expires every 3 months to [epic] The SSL certificate for Beta cluster domains fails to be renewed in time.Oct 17 2021, 10:04 PM

AlexisJazz added subtasks: T293251: The certificate for upload.wikimedia.beta.wmflabs.org expired on October 9, 2021., T262816: The certificate for en.wikipedia.beta.wmflabs.org expired on 2020-09-14, T257968: Certificate for *.beta.wmflabs.org has expired (July 2020), T262806: Beta cluster certificates have expired (September 2020).

RhinosF1 renamed this task from [epic] The SSL certificate for Beta cluster domains fails to be renewed in time to [epic] The SSL certificate for Beta cluster domains fails to properly renew & deploy.Oct 17 2021, 10:21 PM

Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptOct 17 2021, 10:21 PM

Maintenance_bot added a project: SRE.Oct 17 2021, 10:45 PM

AlexisJazz added a project: Epic.Oct 18 2021, 12:46 PM

Dzahn triaged this task as High priority.Oct 18 2021, 6:26 PM

Dzahn lowered the priority of this task from High to Medium.Oct 18 2021, 6:38 PM

Dzahn added a project: Quality-and-Test-Engineering-Team.Oct 18 2021, 6:43 PM

Reedy added a subtask: T296000: *.beta.wmflabs.org Certificate has expired (November 2021 edition).Nov 18 2021, 4:45 PM

AlexisJazz added a subtask: T296113: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021..Nov 19 2021, 9:52 PM

RhinosF1 removed a subtask: T296113: The certificate for upload.wikimedia.beta.wmflabs.org expired on November 18, 2021..Nov 19 2021, 9:53 PM

Urbanecm closed subtask T293070: upload.wikimedia.beta.wmflabs.org certificate expired (October 2021) as Resolved.Nov 19 2021, 10:02 PM

AlexisJazz added a subtask: T301995: The certificate for upload.wikimedia.beta.wmflabs.org expired on February 16, 2022..Feb 17 2022, 3:41 PM

MatthewVernon closed subtask T301995: The certificate for upload.wikimedia.beta.wmflabs.org expired on February 16, 2022. as Resolved.Feb 21 2022, 2:06 PM

MatthewVernon mentioned this in T301995: The certificate for upload.wikimedia.beta.wmflabs.org expired on February 16, 2022..

dom_walden added a subtask: T306492: *.beta.wmflabs.org Certificate has expired (April 2022 edition).Apr 20 2022, 8:31 AM

bd808 added a parent task: T273956: acme-chief sometimes doesn't refresh certificates because it ignores SIGHUP.Apr 20 2022, 2:18 PM

Jrbranaa moved this task from Inbox to Test Infrastructure on the Quality-and-Test-Engineering-Team board.Apr 29 2022, 4:30 PM

TheresNoTime added a subtask: T310957: *.beta.wmflabs.org Certificate has expired (June 2022 edition).Jun 19 2022, 10:33 AM

TheresNoTime closed subtask T310957: *.beta.wmflabs.org Certificate has expired (June 2022 edition) as Resolved.Jun 20 2022, 5:17 PM

Aklapper closed subtask T306492: *.beta.wmflabs.org Certificate has expired (April 2022 edition) as Resolved.Jul 3 2022, 1:46 PM

Aklapper closed subtask T296000: *.beta.wmflabs.org Certificate has expired (November 2021 edition) as Resolved.

Aklapper added a project: User-AKlapper.

We got alerts about the Beta Cluster cert being close to expiry (T311457#8147086) so I again ran:

$ touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service

On deployment-cache-{text,upload}06.

RhinosF1 mentioned this in T311457: Set up health checks for function-* services on Beta Cluster.Aug 12 2022, 7:26 AM

Can someone do the thing again? It expired today.

Please Traffic take a look at this problem. I have solved the immediate issue.

• karapayneWMDE subscribed.Oct 17 2022, 7:57 AM

Michael subscribed.Oct 17 2022, 8:03 AM

The current deployment-prep instances are pretty far from our current production environment, ats-tls is not longer maintained nor ATS 8.x for the backend instance.

I've created T320930 to track this

taavi mentioned this in T320920: test.wikipedia.beta.wmflabs.org security certificate expired in the last day.Oct 17 2022, 1:17 PM

taavi merged a task: T320920: test.wikipedia.beta.wmflabs.org security certificate expired in the last day.

taavi added subscribers: Jeff_G, taavi, Jdforrester-WMF.