'helm_release_status' metric not found in k8s-mlserve
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Apr 13 2023, 9:03 AM

Description

I'm going through all the pint errors and the following error is reported (link to AlertLintProblem)

Pint reporter promql/series found problem(s) in /srv/alerts/k8s-mlserve/team-sre_kubernetes-generic.yaml: prometheus "k8s-mlserve" at http://127.0.0.1:9909/k8s-mlserve didn't have any series for "helm_release_status" metric in the last 1w

After a chat with @elukey he pointed me to T323706: wikikube LIST secrets latency which introduced the metric; then the current question I have is whether it is expected for k8s-mlserve to not carry the metric, or we could expand helm status metrics to run on all k8s cluster, or some other solution I can't see right now? What do you think @JMeybohm ? Thank you!

Details

	Subject	Repo	Branch	Lines +/-
	Install helm-state-metrics by default on all clusters	operations/deployment-charts	master	+2 -9

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		fgiunchedi	T309182 Evaluate 'pint' for Prometheus alerts
		Resolved		JMeybohm	T334647 'helm_release_status' metric not found in k8s-mlserve

Event Timeline

fgiunchedi created this task.Apr 13 2023, 9:03 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 13 2023, 9:03 AM

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Apr 13 2023, 9:03 AM

fgiunchedi edited projects, added SRE Observability (FY2022/2023-Q4); removed SRE Observability (FY2022/2023-Q3).Apr 14 2023, 8:20 AM

There is no reason not to deploy helm-state-metrics to all clusters, I just left that as an decision for the cluster operators to make rather than deploying it everywhere.

fgiunchedi updated the task description. (Show Details)Apr 14 2023, 8:46 AM

In T334647#8781170, @JMeybohm wrote:

There is no reason not to deploy helm-state-metrics to all clusters, I just left that as an decision for the cluster operators to make rather than deploying it everywhere.

Thank you, that makes sense! Do you see any problem with deploying helm-state-metrics to all clusters? Alternatively we could select specific instances via deploy-tag, though that might be more brittle (?) Finally we could ignore the "missing metrics" lint error too

I don't see any problem deploying it to all clusters. @BTullis, @jhathaway wdyt?
For reference: This is what we're talking about: https://gerrit.wikimedia.org/g/operations/software/helm-state-metrics

@JMeybohm sound sensible to me

fgiunchedi moved this task from Doing to Radar on the User-fgiunchedi board.May 23 2023, 8:22 AM

Change 922474 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Install helm-state-metrics by default on all clusters

https://gerrit.wikimedia.org/r/922474

gerritbot added a project: Patch-For-Review.May 23 2023, 9:10 AM

Change 922474 merged by jenkins-bot:

[operations/deployment-charts@master] Install helm-state-metrics by default on all clusters

https://gerrit.wikimedia.org/r/922474

I've deployed helm-state-metrics to all clusters and the LintProblem alerts are gone.
Here is a link to the helm releases dashboard for the record: https://grafana.wikimedia.org/d/UT4GtK3nz/helm-releases

JMeybohm closed this task as Resolved.May 23 2023, 3:28 PM

JMeybohm claimed this task.

Maintenance_bot removed a project: Patch-For-Review.May 23 2023, 3:30 PM

fgiunchedi awarded a token.May 24 2023, 7:57 AM

lmata awarded a token.May 30 2023, 12:55 PM

lmata moved this task from Inbox to Done on the SRE Observability (FY2022/2023-Q4) board.

lmata removed a project: Observability-Alerting.