Page MenuHomePhabricator

Move ToolsDB Grafana dashboard to grafana.wmcloud.org
Closed, ResolvedPublic

Description

We have a very useful Grafana dashboard for ToolsDB at https://grafana-labs.wikimedia.org/d/000000273/tools-mariadb

We want to migrate it to the new Grafana instance at https://grafana.wmcloud.org

While we do it, we should fix a few queries that are still filtering for project=clouddb-services and are not showing the new servers in the tools project (tools-db-1 and tools-db-2).

I would also rename it from Tools MariaDB to ToolsDB MariaDB because I've been typing toolsdb in the search too many times 😛

Event Timeline

fnegri triaged this task as Medium priority.Apr 27 2023, 10:53 AM
fnegri changed the task status from Open to In Progress.May 4 2023, 12:51 PM
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2022/2023-Q4) board.

I have moved the dashboard to https://grafana.wmcloud.org/d/PTtEnEyVk/toolsdb-mariadb and added a link from the old dashboard to the new location.

Some panels are not working but they were already broken in the old dashboard, that is because some metrics changed with the upgrade to MariaDB 10.4.

I will try to fix all the panels (or remove the ones that cannot be fixed) before resolving this task.

I manually edited the dashboard JSON file and replaced all the remaining references to the old project name clouddb-services with the new project name tools.

I removed from the dashboard the panel "Monitoring queries latency" that was displaying an empty chart. I'm confused about this panel because I have a vague memory it was showing data before the upgrade from MariaDB 10.1 to 10.4, but at the same time I don't understand how http_request_duration_microseconds can be a metric related to SQL query latency.

I also removed a few other panels showing empty charts (e.g. "InnoDB Purge Lags") that are based on InnoDB metrics that according to this page are only available in MariaDB 10.5.

List of the removed panels and the metrics they were based on:

  • Monitoring queries latency
    • http_request_duration_microseconds
  • InnoDB Checkpoint Age
    • mysql_global_status_innodb_checkpoint_age
    • mysql_global_status_innodb_checkpoint_max_age
  • Innodb Purge Lag
    • mysql_global_status_innodb_history_list_length
  • Change Buffer
    • mysql_global_status_innodb_ibuf_merged_inserts
    • mysql_global_status_innodb_ibuf_merges
  • Adaptive Hash and other memory usage
    • mysql_global_status_innodb_mem_adaptive_hash
    • mysql_global_status_innodb_mem_dictionary
  • InnoDB Semaphores
    • mysql_global_status_innodb_mutex_os_waits
    • mysql_global_status_innodb_mutex_spin_rounds