Thanos: support multiple ruler instances
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	herron
	Sep 30 2025, 5:12 PM

Description

Today we have a single thanos ruler instance, however it would be quite useful to be able to deploy certain recording rules with a unique external label so they are easier to find/manage/cleanup in thanos storage. The immediate use case for this is piloting recording rules using production metrics

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
thanos-rule: add pilot instance	operations/puppet	production	+66 -10
profile::thanos::query::store_config: add define	operations/puppet	production	+35 -6
thanos-rule: add support for multiple instances	operations/puppet	production	+70 -44
ThanosRecordingRuleGaps: update thanos-rule to thanos-rule@main	operations/alerts	master	+5 -5

Customize query in gerrit

Event Timeline

herron created this task.Sep 30 2025, 5:12 PM

herron changed the task status from Open to In Progress.

herron triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 30 2025, 5:12 PM

herron moved this task from Inbox to FY2025/2026-Q1 on the SRE Observability board.Sep 30 2025, 5:12 PM

herron edited projects, added SRE Observability (FY2025/2026-Q1); removed SRE Observability.

Change #1192209 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] thanos-rule: add pilot instance

https://gerrit.wikimedia.org/r/1192209

Change #1188441 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] thanos-rule: add support for multiple instances

https://gerrit.wikimedia.org/r/1188441

Change #1188441 merged by Herron:

[operations/puppet@production] thanos-rule: add support for multiple instances

https://gerrit.wikimedia.org/r/1188441

Change #1197326 had a related patch set uploaded (by Herron; author: Herron):

[operations/alerts@master] ThanosRecordingRuleGaps: update thanos-rule to thanos-rule@main

https://gerrit.wikimedia.org/r/1197326

Change #1197326 merged by jenkins-bot:

[operations/alerts@master] ThanosRecordingRuleGaps: update thanos-rule to thanos-rule@main

https://gerrit.wikimedia.org/r/1197326

Change #1197669 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] profile::thanos::query::store_config: add define

https://gerrit.wikimedia.org/r/1197669

Change #1197669 merged by Herron:

[operations/puppet@production] profile::thanos::query::store_config: add define

https://gerrit.wikimedia.org/r/1197669

Change #1192209 merged by Herron:

[operations/puppet@production] thanos-rule: add pilot instance

https://gerrit.wikimedia.org/r/1192209

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2025, 8:31 PM

We now have two thanos rule instances running, "main" (the pre-existing instance) and a new instance called "pilot"

Each instance is configured to attach a unique external label recorder=thanos-rule@main, recorder=thanos-rule@pilot etc.

titan1001:~$ systemctl status thanos-rule@pilot
● thanos-rule@pilot.service - Thanos rule (instance pilot)
     Loaded: loaded (/lib/systemd/system/thanos-rule@pilot.service; static)
     Active: active (running) since Thu 2025-10-23 20:04:17 UTC; 41min ago

I think we're good to go here!

Thanos: support multiple ruler instancesClosed, ResolvedPublicActions

Description

Details

Event Timeline

Thanos: support multiple ruler instances
Closed, ResolvedPublic
Actions