Alert on Thanos sidecar not uploading blocks
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Oct 15 2020, 4:05 PM

Description

Today for unrelated reasons I noticed that Thanos sidecar on PoPs wasn't uploading blocks even though it should have post-migration in T243057.

We must alert when Thanos sidecar isn't configured for upload, since that's not a normal condition now. In practice I think alerting on absent(thanos_shipper_uploads_total) for each Thanos sidecar instance should be enough

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	prometheus: add ThanosSidecarUploadFailure to prometheus/ops	operations/puppet	production	+18 -0
	profile: add alerts for Thanos sidecar not uploading or failing to do so	operations/puppet	production	+30 -1

Customize query in gerrit

Related Objects

Mentioned Here: T243057: Move Prometheus off eqsin/ulsfo/esams bastions

Event Timeline

fgiunchedi created this task.Oct 15 2020, 4:05 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2020, 4:05 PM

Change 634475 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: add alerts for Thanos sidecar not uploading or failing to do so

https://gerrit.wikimedia.org/r/634475

gerritbot added a project: Patch-For-Review.Oct 16 2020, 8:59 AM

• lmata moved this task from Inbox to In progress on the observability board.Oct 19 2020, 3:21 PM

Change 634475 merged by Filippo Giunchedi:
[operations/puppet@production] profile: add alerts for Thanos sidecar not uploading or failing to do so

https://gerrit.wikimedia.org/r/634475

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2020, 8:10 AM