Interesting! Since on buster there's an implicit upgrade of mysqld-exporter to 0.11, some of the innodb-related performance schema options need to be enabled? e.g. --collect.info_schema.innodb_metrics although just a guess as I don't know where all metrics needed live

re: "monitoring queries latency" the expression needs to be changed like this (i.e. to handle multiple handlers)

http_request_duration_microseconds{instance="$server:$port", handler=~"(prometheus|metrics)",quantile="0.5"}

The innodb variables, as kinda expected, are failing due to the fact that they've been removed upstream: https://jira.mariadb.org/browse/MDEV-18582
I haven't checked 100% of the ones we used, but I have randomly checked some of the ones we have failing on some of the graphs, and they are all gone.

Those graphs aren't super critical, so we can probably live without them on 10.3 anyways.

In T231190#5437423, @fgiunchedi wrote:

re: "monitoring queries latency" the expression needs to be changed like this (i.e. to handle multiple handlers)

http_request_duration_microseconds{instance="$server:$port", handler=~"(prometheus|metrics)",quantile="0.5"}

Thanks! Would that change break the existing 10.1 hosts?
If that happens, how can we solve the issue that we might have different things depending on the version, as the migration from 10.1 to 10.3 will be done slowly and the will co-exist for months.
Can you help us get this working? :)

In T231190#5437489, @Marostegui wrote:

The innodb variables, as kinda expected, are failing due to the fact that they've been removed upstream: https://jira.mariadb.org/browse/MDEV-18582
I haven't checked 100% of the ones we used, but I have randomly checked some of the ones we have failing on some of the graphs, and they are all gone.

Those graphs aren't super critical, so we can probably live without them on 10.3 anyways.

In T231190#5437423, @fgiunchedi wrote:

re: "monitoring queries latency" the expression needs to be changed like this (i.e. to handle multiple handlers)

http_request_duration_microseconds{instance="$server:$port", handler=~"(prometheus|metrics)",quantile="0.5"}

Thanks! Would that change break the existing 10.1 hosts?
If that happens, how can we solve the issue that we might have different things depending on the version, as the migration from 10.1 to 10.3 will be done slowly and the will co-exist for months.
Can you help us get this working? :)

For sure! The metric above will work for both versions of mysqld-exporter (not mysqld, since the metric belongs to the exporter itself) because one has handler=prometheus and the buster version has handler=metrics.

In general when metrics are renamed you can either add a brand new expression to the panel with the new query, or use old_query or new_query2 in the same expression to evaluate one or the other. Once the migration is complete then old_query can be removed when needed.

Hope that helps! Let us know when you have more questions

Mentioned in SAL (#wikimedia-operations) [2019-08-26T13:21:37Z] <marostegui> Change MySQL.monitoring queries latency graph parameters to support buster+mariadb 10.3 - T231190