| • DAlangi_WMF | |
| Feb 23 2026, 12:10 PM |
| Restricted File | |
| Mar 10 2026, 10:48 AM |
| Restricted File | |
| Mar 10 2026, 10:48 AM |
In T415035: Create prometheus recording rules for MediaWiki CentralAuth dashboard metrics, recording rules were created to help speed up: https://grafana.wikimedia.org/d/fd0c8c5e-3b26-4a90-bca2-a2504de1badc/mediawiki-and-centralauth-sessions.
This task is about using the recording rules created in the task above and observing whether the dashboard can now load 7 days, 14 days, and 30+ days of data relatively quickly, rather than timing out.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • DAlangi_WMF | T418123 Improve CentralAuth dashboard in Grafana to load 30 days without timeout | |||
| Open | tappof | T415035 Create prometheus recording rules for MediaWiki CentralAuth dashboard metrics |
Just checked the https://grafana.wikimedia.org/d/fd0c8c5e-3b26-4a90-bca2-a2504de1badc/mediawiki-and-centralauth-sessions dashboard, and it looks like @Krinkle already started making use of the recording rules (for session reads metric: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1233214).
I've made adjustments to the dashboard to use the recording rules for session write metrics at: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1236233. From a quick browse, it looks like the dashboard has already gotten a little faster (even for 30 days' worth of data).
I'll poke the team to try the dashboard again and see if there are any improvements.
Change #1245279 had a related patch set uploaded (by Krinkle; author: Derick Alangi):
[mediawiki/extensions/CentralAuth@master] CentralAuthSessionManager: Don't split read/write metrics by wiki
Change #1245282 had a related patch set uploaded (by Krinkle; author: Derick Alangi):
[mediawiki/core@master] Session: Remove `wiki` label from session store metrics
Change #1245279 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] CentralAuthSessionManager: Don't split read/write metrics by wiki
Change #1245282 merged by jenkins-bot:
[mediawiki/core@master] Session: Remove `wiki` label from session store metrics
I believe we can move this to "To be verified in production" for a few weeks and see if situations improve with new data. Also, once the dashboard starts responding faster, we can maybe undo/remove the recording rules and see what happens after that, Cc @Krinkle
We've done what we can do on the MW/CentralAuth side so that only the data we need gets stored (removed the wiki label, which increased data cardinality significantly) in Prometheus. We've also spent some time polishing up the dashboard so that only what is needed gets plotted. Together with the recording rules, the dashboard should respond faster in a few weeks.
I can also observe that from 0 - 14 days (2 weeks), the graphs are already doing well, we just need to get to 30 days, which is the amount of time span we go back during chores.
| {F72791182} | {F72791181} |
You mean https://grafana.wikimedia.org/d/000000004/authentication-metrics, right? Yes, I think we can optimize it a little because at 30 days, it takes some time to load the panels, and occasionally, a few panels time out.
I'll file a task about it.
Filed T419510: Improve authentication metrics dashboard to load more than 30 days without timeout.
I'll resolve this now. No more work left to do here and the dashboard is already doing well. We can re-open after a few months (if we run into slowness for any reason).