Page MenuHomePhabricator
Create Task
Maniphest T190978

Update ResourceLoader dashboard to query varnishrls data from Prometheus instead
Closed, ResolvedPublic
Actions

Description

Per T184942#4083351 and T184942#4087564, I think we now have all the data we need in Prometheus.

There is also the https://grafana.wikimedia.org/dashboard/db/resourceloadermodule dashboard but it only uses MediaWiki/ResourceLoader internal metrics that aren't affected by the move the Prometheus right now. Only the Varnish metrics are being moved, used by the main and alert dashboards.

Details

SubjectRepoBranchLines +/-
mtail: Add xcachestatus to varnishrlsoperations/puppetproduction+9 -9
prometheus: Add varnishrls aggregation rulesoperations/puppetproduction+3 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krinkle created this task.Mar 28 2018, 11:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2018, 11:17 PM
Krinkle updated the task description. (Show Details)Mar 28 2018, 11:18 PM
Krinkle updated the task description. (Show Details)
• Imarlier triaged this task as Low priority.Apr 2 2018, 8:13 PM
• Imarlier moved this task from Inbox, needs triage to To-do: Goals prioritized current Quarter on the Performance-Team board.
Krinkle moved this task from Inbox to Accepted Enhancement on the MediaWiki-ResourceLoader board.Apr 9 2018, 7:51 PM
Krinkle raised the priority of this task from Low to Medium.May 4 2018, 3:17 AM
Krinkle claimed this task.May 7 2018, 1:25 PM
Krinkle moved this task from To-do: Goals prioritized current Quarter to Doing (old) on the Performance-Team board.
gerritbot subscribed.May 10 2018, 4:57 PM

Change 432090 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] prometheus: Add varnishrls aggregation rules

https://gerrit.wikimedia.org/r/432090

gerritbot added a project: Patch-For-Review.May 10 2018, 4:57 PM
gerritbot added a comment.May 11 2018, 7:51 AM

Change 432090 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: Add varnishrls aggregation rules

https://gerrit.wikimedia.org/r/432090

Krinkle added a subscriber: fgiunchedi.May 12 2018, 12:48 AM
In T190978#4199354, @gerritbot wrote:

Change 432090 merged by Filippo Giunchedi:
<operations/puppet@production>: prometheus: Add varnishrls aggregation rules

@fgiunchedi Thanks. This makes the per-dc stacks much easier without the need to iterate over each data source. Interestingly though, while I still need the aggregated rules for more complex graphs, I did find a way to make the simpler graphs work without the global rules. Namely, Grafana supports a way to mix multiple data sources in a single graph:

Screen Shot 2018-05-12 at 02.40.03.png (704×2 px, 425 KB)

Global (easy!)
Screen Shot 2018-05-12 at 02.40.23.png (342×1 px, 62 KB)
Mixed (not impossible!)
Screen Shot 2018-05-12 at 02.40.14.png (642×1 px, 117 KB)
gerritbot added a comment.May 12 2018, 1:56 AM

Change 432712 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] mtail: Add xcachestatus to varnishrls

https://gerrit.wikimedia.org/r/432712

fgiunchedi added a comment.May 15 2018, 8:25 AM
In T190978#4201938, @Krinkle wrote:
In T190978#4199354, @gerritbot wrote:

Change 432090 merged by Filippo Giunchedi:
<operations/puppet@production>: prometheus: Add varnishrls aggregation rules

@fgiunchedi Thanks. This makes the per-dc stacks much easier without the need to iterate over each data source. Interestingly though, while I still need the aggregated rules for more complex graphs, I did find a way to make the simpler graphs work without the global rules. Namely, Grafana supports a way to mix multiple data sources in a single graph:

Nice! Thanks that's good to know Grafana can do that

gerritbot added a comment.May 22 2018, 2:55 PM

Change 432712 merged by Ema:
[operations/puppet@production] mtail: Add xcachestatus to varnishrls

https://gerrit.wikimedia.org/r/432712

Krinkle mentioned this in T190562: Investigate ResourceLoader INM and 304 regressions.May 23 2018, 2:31 PM
Krinkle removed a project: Patch-For-Review.May 23 2018, 4:23 PM
Krinkle closed this task as Resolved.May 24 2018, 11:58 AM

This is now done. All frontend panels on https://grafana-admin.wikimedia.org/dashboard/db/resourceloader and https://grafana-admin.wikimedia.org/dashboard/db/resourceloader-alerts use Prometheus now. I've also added a datacenter site breakdown option to the main RL dashboard.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL