Page MenuHomePhabricator

Alert on Thanos sidecar not uploading blocks
Closed, ResolvedPublic

Description

Today for unrelated reasons I noticed that Thanos sidecar on PoPs wasn't uploading blocks even though it should have post-migration in T243057.

We must alert when Thanos sidecar isn't configured for upload, since that's not a normal condition now. In practice I think alerting on absent(thanos_shipper_uploads_total) for each Thanos sidecar instance should be enough

Event Timeline

Change 634475 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: add alerts for Thanos sidecar not uploading or failing to do so

https://gerrit.wikimedia.org/r/634475

Change 634475 merged by Filippo Giunchedi:
[operations/puppet@production] profile: add alerts for Thanos sidecar not uploading or failing to do so

https://gerrit.wikimedia.org/r/634475

fgiunchedi claimed this task.

This is complete!

Change 719126 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: add ThanosSidecarUploadFailure to prometheus/ops

https://gerrit.wikimedia.org/r/719126

Change 719126 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: add ThanosSidecarUploadFailure to prometheus/ops

https://gerrit.wikimedia.org/r/719126