Page MenuHomePhabricator
Create Task
Maniphest T366579

[infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire
Closed, ResolvedPublic
Actions

Description

This will allow us to not forget to renew it

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
toolforge::prometheus: add exporter for the k8s cert expiryoperations/puppetproduction+48 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
toolforge: add prometheus k8s cert expiry alertrepos/cloud/toolforge/alerts!24dcaroadd_prometheus_k8s_certmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

dcaro created this task.Jun 4 2024, 10:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2024, 10:51 AM
dcaro triaged this task as High priority.Jun 4 2024, 10:51 AM
dcaro changed the task status from Open to In Progress.Jun 4 2024, 1:26 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 10) board.
dcaro edited projects, added Toolforge (Toolforge iteration 11); removed Toolforge (Toolforge iteration 10).Jun 4 2024, 2:55 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 11) board.
taavi subscribed.Jun 4 2024, 3:01 PM

https://samber.github.io/awesome-prometheus-alerts/rules.html#rule-kubernetes-1-32

dcaro added a comment.Jun 4 2024, 3:24 PM
In T366579#9860099, @taavi wrote:

https://samber.github.io/awesome-prometheus-alerts/rules.html#rule-kubernetes-1-32

awesome!

dcaro added a comment.Jun 11 2024, 2:30 PM

This seems a bit trickier than just adapting the above query, still investigating how to get that info

dcaro edited projects, added Toolforge (Toolforge iteration 12); removed Toolforge (Toolforge iteration 11).Jul 2 2024, 2:00 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 12) board.
dcaro edited projects, added Toolforge (Toolforge iteration 13); removed Toolforge (Toolforge iteration 12).Jul 17 2024, 11:22 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 13) board.
dcaro edited projects, added Toolforge (Toolforge iteration 14); removed Toolforge (Toolforge iteration 13).Aug 5 2024, 7:39 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 14) board.
dcaro edited projects, added Toolforge (Toolforge iteration 15); removed Toolforge (Toolforge iteration 14).Sep 25 2024, 7:56 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 15) board.
dcaro edited projects, added Toolforge (Toolforge iteration 16); removed Toolforge (Toolforge iteration 15).Oct 15 2024, 7:33 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 16) board.
gerritbot added a comment.Nov 25 2024, 5:43 PM

Change #1097450 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] toolforge::prometheus: add exporter for the k8s cert expiry

https://gerrit.wikimedia.org/r/1097450

gerritbot added a project: Patch-For-Review.Nov 25 2024, 5:43 PM
dcaro moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 16) board.Nov 25 2024, 6:17 PM
gerritbot added a comment.Nov 28 2024, 2:06 PM

Change #1097450 merged by David Caro:

[operations/puppet@production] toolforge::prometheus: add exporter for the k8s cert expiry

https://gerrit.wikimedia.org/r/1097450

Maintenance_bot removed a project: Patch-For-Review.Nov 28 2024, 2:30 PM
CodeReviewBot added a project: Patch-For-Review.Nov 28 2024, 4:50 PM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/24

toolforge: add prometheus k8s cert expiry alert

CodeReviewBot added a comment.Nov 28 2024, 4:58 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/24

toolforge: add prometheus k8s cert expiry alert

dcaro added a comment.Nov 28 2024, 5:06 PM

Up and running!

image.png (355×711 px, 43 KB)

dcaro closed this task as Resolved.Nov 28 2024, 5:06 PM
dcaro moved this task from In Review to Done on the Toolforge (Toolforge iteration 16) board.
Maintenance_bot removed a project: Patch-For-Review.Nov 28 2024, 5:30 PM
taavi merged a task: T284310: Monitor Toolforge external kubernetes certificate expiry.Sep 23 2025, 5:50 PM
taavi added a subscriber: • Bstorm.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits