Another idea for better dashboarding: show vertical lines for events other than deployments, e.g. puppet merges

In T178690#3706215, @fgiunchedi wrote:

Another idea for better dashboarding: show vertical lines for events other than deployments, e.g. puppet merges

I agree, in a setup I've done in the past I had all annotations saved to an Elasticsearch, to allow it to scale quite easily. The smart thing to do here would be to have those other annotations tagged so that each dashboard could maybe by default only show related annotations and optionally show all of them. As a practical example, puppet merges could be tagged and by default in a traffic dashboard we show only traffic-tagged merges, but optionally you can show all of them, because sometimes things correlates in unpredictable ways...

I'd also like to add to:

There are three main components we can drill down/up: site/cluster/host

that it would be nice to be able to both show together all said metric for all hosts in a cluster or alternatively all said metric aggregated for all hosts in a cluster. This add a bit of complexity given that you need to know how to aggregate based on the metric (sum, average, etc...).
For the simple cases that just require a sum, also a stack graph can achieve the same, and for the average case a non-stacked graph gives you an idea of it, but they become quickly unreadable with the number of hosts/metrics plotted.

T180784 has some interesting discussion as well.

@ori recently sent his thoughts about this to the ops list, and I found it a very eloquent description of the issues I was thinking of too. His full email was:

Ganglia may have been a buggy and crufty, but when it was accessible anyone could see a high-level overview of Wikimedia's operational metrics at a glance by browsing to https://ganglia.wikimedia.org/.

This was extremely useful for spotting and troubleshooting problems. And the fact that anyone was invited to have a look was a powerful demonstration of what Wikimedia is and what makes it special.

Operational metrics today may be more comprehensive, more accurate, and/or more reliable, but they are not more discoverable.
The list of featured dashboards in https://grafana.wikimedia.org/ is not well-organized. There is a curious mix of important and unimportant dashboards, which are not grouped in any meaningful way. Some dashboards that should be featured aren't, and some featured dashboards shouldn't be.

The names of the dashboards are often obscure, vague, or confusing. What is "Production Logging"? Why do "Prometheus DC overview" and "Prometheus global overview" have "Prometheus" in their name?

Some of the top-level links refer to teams ("Team TCB"), others to topics ("Performance Metrics"), others to specific services ("Swift").

So, this a plea for a good landing page for operational metrics. I'd really love to see a curated selection of dashboards grouped according to some sensible taxonomy, their names standardized and revised for clarity. I think the time investment will pay for itself in time saved when debugging issues and on-boarding new folks.

I think there are a few different dimensions to this problem:

• Naming (Varnish vs. Traffic vs. HTTPS, "Prometheus" prefixes, etc.)
• Organization/hierachy (AQS has dashboards named as "AQS :: Cassandra :: CF :: Latency/rate Copy" for instance)
• Similar to the above: 1:N tagging/hashtags
• Navigation (drilling up/down, featured dashboards/frontpage)
• Discoverability, which is an artifact of all to the above

We need the following new dashboards / URLs (noticed as part of T183873):

• service cluster A overview (single link) (replace link on https://wikitech.wikimedia.org/wiki/Mathoid)
• service cluster B overview (single link) (replace link on https://wikitech.wikimedia.org/wiki/Citoid)
• parsoid (canary) machines (replace link " Watch the canary machines on ganglia for eqiad and codfw" on https://wikitech.wikimedia.org/wiki/Parsoid#Deploying_the_latest_version_of_Parsoid)
• maps cluster / maps-varnish cluster (replace links on https://wikitech.wikimedia.org/wiki/Maps)
• bits caches, apache cluster, fatals in last hour as used on https://www.mediawiki.org/wiki/Performance_profiling_for_Wikimedia_code#Ganglia

In T178690#3891946, @Dzahn wrote:

We need the following new dashboards / URLs (noticed as part of T183873):

• service cluster A overview (single link) (replace link on https://wikitech.wikimedia.org/wiki/Mathoid)

Mathoid is on SCB, not SCA. only zotero is on SCA. In any case, are these 2 sufficient ?

[eqiad] https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?orgId=1&var-
datasource=eqiad%20prometheus%2Fops&var-cluster=scb&var-instance=All

[codfw] https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?orgId=1&var-datasource=codfw%20prometheus%2Fops&var-cluster=scb&var-instance=All

• service cluster B overview (single link) (replace link on https://wikitech.wikimedia.org/wiki/Citoid)

Same as above.

• parsoid (canary) machines (replace link " Watch the canary machines on ganglia for eqiad and codfw" on https://wikitech.wikimedia.org/wiki/Parsoid#Deploying_the_latest_version_of_Parsoid)

I see no differentiation for "canary" in the links in the wikitech page. So I am guessing it was a mental process for the parsoid deployer. In that case, following the pattern above we have

[codfw] https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?orgId=1&var-datasource=codfw%20prometheus%2Fops&var-cluster=parsoid&var-instance=All

[eqiad] https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=parsoid&var-instance=All

• maps cluster / maps-varnish cluster (replace links on https://wikitech.wikimedia.org/wiki/Maps)

The maps-varnish does not exist anymore (T164608) so there is nothing to do about that. For maps itself, following the pattern above

[eqiad] https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=maps&var-instance=All

[codfw] https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=maps&var-instance=All

The question

All of the above are practically the exact same link with a bit of a tweaking to set the cluster and datacenter. Should we follow the more brittle approach of updating every page with the cluster+DC specific link or should we go for the more robust approach of just using the base https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?orgId=1 and let the user figure it out ? At least for the DC part I am pretty sure the latter, I am not so sure about the cluster though.

I've put together a sample dashboard to play around with some concepts/ideas emerged in this task at https://grafana.wikimedia.org/dashboard/db/dashboard-redesign-proposal . Notably missing is the navigation story among different dashboards, but tl;dr it would be based on dashboards tags to create dropdowns. Which grouping/dropdown menus make sense is still TBD.

The dashboard has been put together by me by point-and-click, though the idea is to have the same dashboard generated by code and thus making it simple to create multiple consistent dashboards for different services and purposes.

As discussed in the monitoring meeting here some feedback:

• while the limit on the number of rows/panels/metrics is understandable, it could make harder to make generic dashboard for some services, and splitting them into multiple dashboards might made harder their discoverability. One option is to exclude the rows/panels/metrics that are hidden by default from the "limit".
• I'm not sure if we should come up with some best practice for the values to show near a single label (min/max/avg/current) and whether they should be inline/as table/as table on the right. Different graphs might need different values/layout based on the data shown.
• I'm personally a big fan of the shared crosshair, maybe we could set it on by default.

In T178690#4168673, @Volans wrote:

As discussed in the monitoring meeting here some feedback:

• while the limit on the number of rows/panels/metrics is understandable, it could make harder to make generic dashboard for some services, and splitting them into multiple dashboards might made harder their discoverability. One option is to exclude the rows/panels/metrics that are hidden by default from the "limit".

Agreed. From what I understood anyway, the limit is suggested for performance reasons and given "hidden" rows do not get evaluated at dashboard load time but rather on "unhiding".

• I'm not sure if we should come up with some best practice for the values to show near a single label (min/max/avg/current) and whether they should be inline/as table/as table on the right. Different graphs might need different values/layout based on the data shown.

I am very ambivalent on the legend as well. I tend to create it, but I have no rule yet and rather play it by ear. I 'd say we say SHOULD instead of MUST for any kind of guideline here and leave it to the graph creator.

• I'm personally a big fan of the shared crosshair, maybe we could set it on by default.

+1

• I also think we should add a RED/4 golden signals method example to the proposal before we make it to a template. Granted SRE graph will probably use the USE method (pun intended) but still, it'd be great to have an example of that as well

Thanks for the feedback!

In T178690#4171433, @akosiaris wrote:

In T178690#4168673, @Volans wrote:

As discussed in the monitoring meeting here some feedback:

• while the limit on the number of rows/panels/metrics is understandable, it could make harder to make generic dashboard for some services, and splitting them into multiple dashboards might made harder their discoverability. One option is to exclude the rows/panels/metrics that are hidden by default from the "limit".

Agreed. From what I understood anyway, the limit is suggested for performance reasons and given "hidden" rows do not get evaluated at dashboard load time but rather on "unhiding".

I suggested the limit of 5/6 rows per dashboard to avoid too much information per dashboard, though performance is also a concern of course. I think having one "overview" dashboard that is canonical and one/more dashboards for drilldown(s) could work. I expanded on this point in the dashboard example.

• I'm not sure if we should come up with some best practice for the values to show near a single label (min/max/avg/current) and whether they should be inline/as table/as table on the right. Different graphs might need different values/layout based on the data shown.

I am very ambivalent on the legend as well. I tend to create it, but I have no rule yet and rather play it by ear. I 'd say we say SHOULD instead of MUST for any kind of guideline here and leave it to the graph creator.

Indeed, my guideline generally is to display whichever summary aids in issue debugging e.g. max for utilization, max/total for errors, min for availability, etc. I'd say ideally no more than two summaries per graph, added an explanation to the dashboard sample for this too.

• I'm personally a big fan of the shared crosshair, maybe we could set it on by default.

+1

+1 too, added to the dashboard

• I also think we should add a RED/4 golden signals method example to the proposal before we make it to a template. Granted SRE graph will probably use the USE method (pun intended) but still, it'd be great to have an example of that as well

Agreed, I'll try to come up with a sample dashboard for those too. Our USE cases (ha ha) I think depend a whole lot if we're diagnosing performance problems (USE) and/or looking at a service as a whole (RED/4GS)

Change 442301 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] WIP grafana: host overview dashboard as code

https://gerrit.wikimedia.org/r/442301

I coded a strawman using grafanalib at https://gerrit.wikimedia.org/r/c/operations/puppet/+/442301 and looks good to me so far, please take a look too. I'll expand it to multiple dashboards and use cases as well.

In T178690#4319336, @fgiunchedi wrote:

I coded a strawman using grafanalib at https://gerrit.wikimedia.org/r/c/operations/puppet/+/442301 and looks good to me so far, please take a look too. I'll expand it to multiple dashboards and use cases as well.

Note that "dashboards as code" is in scope for T171482: Programmatic generation of grafana dashboards not for this task, which is about dashboard organization in general instead.

Change 444219 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] grafana: use host-overview in favour of server-board for featured dashboard

https://gerrit.wikimedia.org/r/444219

In T178690#3890148, @faidon wrote:

@ori recently sent his thoughts about this to the ops list, and I found it a very eloquent description of the issues I was thinking of too. His full email was:

Ganglia may have been a buggy and crufty, but when it was accessible anyone could see a high-level overview of Wikimedia's operational metrics at a glance by browsing to https://ganglia.wikimedia.org/.

This was extremely useful for spotting and troubleshooting problems. And the fact that anyone was invited to have a look was a powerful demonstration of what Wikimedia is and what makes it special.

Operational metrics today may be more comprehensive, more accurate, and/or more reliable, but they are not more discoverable.
The list of featured dashboards in https://grafana.wikimedia.org/ is not well-organized. There is a curious mix of important and unimportant dashboards, which are not grouped in any meaningful way. Some dashboards that should be featured aren't, and some featured dashboards shouldn't be.

The names of the dashboards are often obscure, vague, or confusing. What is "Production Logging"? Why do "Prometheus DC overview" and "Prometheus global overview" have "Prometheus" in their name?

Some of the top-level links refer to teams ("Team TCB"), others to topics ("Performance Metrics"), others to specific services ("Swift").

So, this a plea for a good landing page for operational metrics. I'd really love to see a curated selection of dashboards grouped according to some sensible taxonomy, their names standardized and revised for clarity. I think the time investment will pay for itself in time saved when debugging issues and on-boarding new folks.

I think there are a few different dimensions to this problem:

• Naming (Varnish vs. Traffic vs. HTTPS, "Prometheus" prefixes, etc.)
• Organization/hierachy (AQS has dashboards named as "AQS :: Cassandra :: CF :: Latency/rate Copy" for instance)
• Similar to the above: 1:N tagging/hashtags
• Navigation (drilling up/down, featured dashboards/frontpage)
• Discoverability, which is an artifact of all to the above

I agree with the above, with the intent of understanding better what's in grafana over the last couple of days I went through the current list of dashboards (~350) and took some notes:

• Some dashboards are user/private/temporary
• "Rotting" dashboards, e.g. metrics disappeared, dashboard not functional, etc.
• I've tagged with obsolete dashboards that seem like they can be deleted (e.g. are graphite-based but we have their prometheus equivalent)
• I've tagged with needs-review dashboards that need some action to decide whether to delete or keep (and eventually update or consolidate in some other dashboard)
• restbase (and restbase staging) dashboards I believe can be deleted for the most part as we're now using their prometheus counterpart (cc @Eevans)
• kubernetes staging dashboards can be folded into their k8s counterparts, since we can vary the datasource to get k8s staging data
• I will remove the "Prometheus" prefix from "prometheus dc-overview / cluster-breakdown / global-overview" and update their navigation/drilldown accordingly
• Some dashboard have a text panel at the top that explains the dashboard a little bit and contains some pointers e.g. to wikitech, this is very nice and we should extend it to all main/important dashboards.

For the featured dashboards list I would say to begin with "four golden signals" for the systems/services that get most user requests: gdns, nginx, varnish, lvs, apache, mediawiki, mysql, swift (likely more, off top of my head and looking at https://wikitech.wikimedia.org/wiki/Category:Wikimedia_infrastructure#/media/File:Infrastructure_overview.png)

In T178690#4403664, @fgiunchedi wrote:

[ ... ]

• restbase (and restbase staging) dashboards I believe can be deleted for the most part as we're now using their prometheus counterpart (cc @Eevans)

{{done}}

Change 444219 merged by Alexandros Kosiaris:
[operations/puppet@production] grafana: use host-overview in favour of server-board for featured dashboard

https://gerrit.wikimedia.org/r/444219

Somewhat related, Grafana upstream has this issue for feedback on dashboard provisioning workflows https://github.com/grafana/grafana/issues/13823

I've generated a list of Grafana dashboards sorted by their last modification time. Details below.

While this isn't a perfect proxy for "is in active use", I think it's probably a pretty good signal.

I suspect we should be trying to find owners for any very old dashboard, and just deleting them if we can't.

sqlite> select d.title, dashboard_id, max(v.created), printf("https://grafana.wikimedia.org/d/%09d/%s", d.id, d.slug) as url from dashboard_version as v left join dashboard as d on d.id = v.dashboard_id group by dashboard_id order by v.created limit 10;

List is here:
https://phabricator.wikimedia.org/P7951

I've just seen a dashboard I use is scheduled for deletion. I don't see the replacement as particularly better and lacking. Could you have a look at how other people are doing those such as https://pmmdemo.percona.com/graph/d/qyzrQGHmk/system-overview They can be downloaded at https://github.com/percona/grafana-dashboards

Jaime, going to have to guess here; are you referring to "Prometheus machine stats" (marked for deletion) vs "Host overview"?

In T178690#4876994, @CDanis wrote:

Jaime, going to have to guess here; are you referring to "Prometheus machine stats" (marked for deletion) vs "Host overview"?

Yes.

In T178690#4877021, @jcrespo wrote:

In T178690#4876994, @CDanis wrote:

Jaime, going to have to guess here; are you referring to "Prometheus machine stats" (marked for deletion) vs "Host overview"?

Yes.

I 've kind of just met the same problem, and had the same immediate reaction as @jcrespo. But the more I am looking at "Host overview" the more I like it. @jcrespo, I think it 's import ant to spell out exactly what you don't like or find lacking in the new dashboard. It does use the "USE" [1] method so it takes a bit to wrap you mind around the methodology, which might or might not be suited for your use case. But in that case we should know.

AFAICT, my gripe is the misc section where it's not at all clear to me what misc: errors is about. I also find that we might need to add some icmp/tcp/udp graphs, perhaps in a by default collapsed row?

[1] http://www.brendangregg.com/usemethod.html

Ah, forgot to update the task, but at the time @jcrespo and @fgiunchedi and I talked, and Jaime's biggest gripe was that iostat-reported "disk IO utilization" is not a very useful metric: it's the fraction of time that at least one oustanding iop was in the disk's queue. On a server that has any load at all, this metric will generally be "100%" all the time; what you actually care about are stats like "queue depth" and "request latencies".

https://www.percona.com/blog/2017/08/28/looking-disk-utilization-and-saturation/ had some good thoughts on the issue as well.

@akosiaris we had some chat about details, I don't mind the USE pattern, but a poor graph using USE doesn't mean it is good, if the chosen metrics are poor, like the above example. Note also they were probably in a worse state before my comments 0:-)

In T178690#4909073, @jcrespo wrote:

@akosiaris we had some chat about details, I don't mind the USE pattern, but a poor graph using USE doesn't mean it is good, if the chosen metrics are poor, like the above example. Note also they were probably in a worse state before my comments 0:-)

Sure. Seems like I was just missing context. Thanks for the update

@fgiunchedi I had a stab at a RED pattern dashboard for mathoid. Let me know what you think.

https://grafana.wikimedia.org/d/000000187/service-mathoid?refresh=1m&orgId=1

I still have on the TODO list to add p50, p90, p99 for latency that is currently missing

It would be nice to make it a little clearer what the intended replacement is (either put it in the task description or the description of the dashboard to be deleted) so one does not have to read through the conversation in this task to know what to update links to.

Mentioned in SAL (#wikimedia-operations) [2023-01-10T13:44:57Z] <godog> delete grafana dashboards from "sre dashboards for deletion" folder - T178690