@Manuel This seems more like a subtask of T314133: Allow recording Wikibase REST API usage for Wikidata monitoring and metrics than the parent task. This task will shortly be completed and I'm assuming discussion on the monitoring and metrics topic will continue in T314133? What do you think?

Works for me, thx!

A Wikibase REST API on Wikidata dashboard has been created.

Lessons learned:

• For these types of graphs, it was important to use the consolidateBy function with the 'sum' function after grouping or summing series together to prevent the average of the grouped/summed series to be displayed.
• You can use graphite queries in graphite queries with a #[A-Z] (i.e. #A, #B, ..., #Z). This only works for queries labelled with a single letter though (e.g. you can't reference a query labelled #Errors).
• You can hide graphite queries from displaying on the graph by hitting the eye icon. This is useful when combined with the previous bullet point.
• Stacked graphs are better for seeing totals of series, non-stacked graphs are better for comparing series.
• Remember to save the dashboard for panel changes to persist. Don't leave it too late to save the dashboard as your session might time out and you will loose your changes when it errors out trying to save.
• You can find the version of graphite being used by going to https://graphite.wikimedia.org/version. This is useful to be able to view the correct version of the docs.

Thank you for creating this and adding your lessons learned!

Do you happen to have a recommended way to drill down into the errors? Say, understanding what 4xx and 5xx codes were returned. Or is that last panel only 5xx and we should see all the data in logstash?

The dashboard has multiple limitations but is good enough for now. Thank you!

@Michael From what I have understood the way Mediawiki reports those makes it impossible to get detail on the kind of error. We do have logstash to dig into 500 errors: https://logstash.wikimedia.org/app/dashboards#/view/c84cd2f0-9b6a-11ed-9fa2-93d8c6813b6b?_g=h@0d13436&_a=h@f1320bf
Does not quite help understanding what kind of 4xx errors happen how often, but we'll get there eventually I assume

We can't view the different 2xx and 3xx responses in grafana as that information isn't recorded with statsd, but we can view the different 4xx and 5xx responses. See Silvan's playpit dashboard for some examples.

Thanks @Ollie.Shotton_WMDE for correcting my fake news!