Page MenuHomePhabricator

toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition)
Open, Stalled, HighPublic

Description

Refresh Toolforge k8s certificates for prometheus and the admission controllers.

For previous work, see T308402. For automating the admission controller certificates, see T292238.

For the admission controller webhooks, rerun the get-cert.sh script similar to the doc, but do not bother the ca-bundle.sh script as that is no longer necessary at all except for local testing. That should inject the secret. To use the secret, delete the appropriate pods in the ingress-admission and registry-admission workspaces to restart them one at a time. Generally the README on the repos for these are the most authoritative docs if in doubt. This is automated now.

For the Prometheus ones, follow the doc on wikitech to recreate the certs.

Event Timeline

dcaro triaged this task as Medium priority.Jun 2 2022, 12:42 PM
dcaro created this task.

@dcaro: Hi, the Due Date set for this open task passed a while ago.
Could you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!

Prometheus got renewed via T338025: [tools] Prometheus k8s cert expired this year, I'll re-use this task for next year's renewal.

taavi renamed this task from toolforge: Refresh certs that are not controlled by kubeadm (mid 2023 edition) to toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition).Sep 27 2023, 7:59 AM
taavi set Due Date to Jun 29 2024, 9:00 PM.
taavi changed Due Date from Jun 29 2024, 9:00 PM to May 29 2024, 9:00 PM.May 10 2024, 1:49 PM
taavi moved this task from Backlog to Ready to be worked on on the Toolforge board.

The current certificates expire on June 1st.

taavi raised the priority of this task from Medium to High.May 10 2024, 1:52 PM

Change #1037888 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] toolforge,prometheus: renew certificate

https://gerrit.wikimedia.org/r/1037888

For the secrets file, this time with puppet7 I had to disable the pre-commit hook under /srv/git/labs/private/.git/hooks or it would not let me commit saying to use the 'fronted puppetserver', but afaik we only have no e puppetserver :/

Change #1037888 merged by David Caro:

[operations/puppet@production] toolforge,prometheus: renew certificate

https://gerrit.wikimedia.org/r/1037888

This is deployed and working, we might want to add an alert warning about certificates "about to expire" to avoid this happening again, but that can be taken care of on monday.

dcaro changed the task status from Open to In Progress.Jun 4 2024, 2:55 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 11) board.
dcaro changed the task status from In Progress to Stalled.Jun 12 2024, 2:10 PM

@dcaro: Hi, the Due Date set for this open task passed a while ago.
Could you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!