I like the idea of having disk usage in the gitlab-runner dasboard too. Although this will be for Trusted Runners only, as Shared Runners in WMCS are not scraped by Prometheus currently.

It's suprisingly hard for me to do that, would enjoy some teachings from others with experience creating dashboards in grafana-rw.

We can pair on Monday if you like. I often use a combination of Grafanas copy & paste function or just copy the raw JSON from an existing dashboard. (so either click on headline -> more -> copy and then on the other dashboard top right + symbol -> paste panel from clipboard. Or just click on headline -> inspect -> panel json). But we can do that together if you like.

In T327435#8564033, @Jelto wrote:

I like the idea of having disk usage in the gitlab-runner dasboard too.
We can pair on Monday if you like.

Sounds good to me:) thank you!

I just found grafana-cloud.wikimedia.org. So we can also try to add dashboards for Shared Runners. There seems to be some metrics available like: https://grafana-cloud.wikimedia.org/d/000000590/instance-details?orgId=1&var-project=gitlab-runners&var-job=node&var-node=runner-1021&from=now-24h&to=now

@Dzahn and I created multiple dashboards for runner disk space usage:

Overview over all Trusted Runners: https://grafana.wikimedia.org/d/Chb-gC07k/gitlab-ci-overview?orgId=1&from=now-7d&to=now&viewPanel=10
Detail for one Runner: https://grafana.wikimedia.org/d/H6fikj0nk/gitlab-runner-detail?orgId=1&refresh=30s&viewPanel=26

And we started a dashboard in Grafana Cloud: https://grafana-cloud.wikimedia.org/d/FrErwP0Vk/gitlab-runner-overview?orgId=1&from=now-7d&to=now

It seems the node exporter for cloud hosts is missing metrics for /var/lib/docker. I'll open a follow up task to get this metric also into prometheus and the dashboard.

Awesome!

follow-up task created at T328972

follow-up task was declined almost immediately and we are now being blocked by that. T328972#8591290