Page MenuHomePhabricator

write some recording rules for queries used in the appserver RED k8s dashboard
Open, HighPublic

Description

The appservers RED k8s dashboard has a lot of useful functionality but several of the queries are pretty heavy for prometheus and take a while to evaluate (especially for longer time intervals).

Those queries should be converted to recording rules, and then after some time later, the dashboard can be edited to use them.

Event Timeline

MLechvien-WMF subscribed.

This appservers RED dashboard got deprecated in favor of https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s, so marking this task as Obsolete.

Feel free to reopen if any action is needed.

CDanis raised the priority of this task from Medium to High.

The appservers RED k8s dashboard makes even heavier queries, and was the trigger of a Thanos outage this week.

MLechvien-WMF renamed this task from write some recording rules for queries used in the appserver RED dashboard to write some recording rules for queries used in the appserver RED k8s dashboard.Jan 16 2026, 4:46 PM
MLechvien-WMF edited projects, added ServiceOps new, Prod-Kubernetes; removed serviceops.
MLechvien-WMF updated the task description. (Show Details)

Thanks and sorry for the wrong triaging.

Renaming this task to include the k8s, and moving it back to the Inbox to consider what action we can take.

@CDanis for the time being I made some really minor changes (eg prevent choosing both DCs, removed auto refresh), until this is picked up properly

o11y will look into adding recording rules for this dashboard.