If we gathered mysql/mariadb stats from the beta cluster, we might be able to spot performance issues on beta, before code gets to production.
I believe to do this we would need to get the prometheus-mysqld-exporter setup on beta cluster DB servers, which I believe at the moment are deployment-db07 and deployment-db08.
We currently do this on production.
The exporter is already running on the db servers. However it's not scraped on any Prometheus instances nor are there any grafana dashboards displaying that info yet.
Who is going tp be realistically be looking at those dashboards regularly enough for them to be helpful to spot issues before they make it to production?
Oh? I ssh'd to deployment-db07 and curl'd localhost:9100/metrics but didn't see any database metrics. What do we need to do to get it scraped?
Who is going tp be realistically be looking at those dashboards regularly enough for them to be helpful to spot issues before they make it to production?
The quality and test engineering team will be looking at these metrics. For sure, the beta server may not be reliable enough or have enough traffic for us to get meaningful performance results. However, we want to try it out at least to see if we can spot performance issues.
Oops, indeed, it's not :/
Someone needs to update the puppet manifests to set it up (most likely adding profile::mariadb::monitor::prometheus to role::mariadb::beta), add the needed mariadb grants and update profile::prometheus::beta to get it scraped.