I 'll try and draft out a possible way out of this. I am adding members of the observability team for input.

So the crux of the issue is that queries like this

sum(rate(container_cpu_user_seconds_total{job="k8s-node-cadvisor", namespace="$service", site="$site", prometheus="$prometheus", pod_name=~"$service.*", container_name=~"$service.*"}[5m]))

will not return results for the cluster that are 1.16+ cause pod_name and container_name doesn't exist anymore. The proposal I got is the following:

• Fetch the JSON for all the dashboards under the Service/ and Kubernetes/ grafana folders.
• Programmatically parse the JSON, and whenever a query like the above is met add an extra query that will just have pod instead of pod_name and container instead of container_name.
• Save the new JSON
• Upload it to grafana (possibly manually but it's not that big a deal, it's like 40 dashboards or so).

This will double the requests to prometheus/thanks for the migration period (which should last roughly as much time as thanos will keep the old data around).

How does the above sound?

In T275618#6856620, @akosiaris wrote:

The proposal I got is the following:

• Fetch the JSON for all the dashboards under the Service/ and Kubernetes/ grafana folders.
• Programmatically parse the JSON, and whenever a query like the above is met add an extra query that will just have pod instead of pod_name and container instead of container_name.
• Save the new JSON
• Upload it to grafana (possibly manually but it's not that big a deal, it's like 40 dashboards or so).

This solution is what we ended up doing for nearly all node exporter metrics in T213708. To this day, nearly every panel on the Host Overview dashboard has an or clause.

The proposed solution SGTM.

In T275618#6856943, @colewhite wrote:

In T275618#6856620, @akosiaris wrote:

The proposal I got is the following:

• Fetch the JSON for all the dashboards under the Service/ and Kubernetes/ grafana folders.
• Programmatically parse the JSON, and whenever a query like the above is met add an extra query that will just have pod instead of pod_name and container instead of container_name.
• Save the new JSON
• Upload it to grafana (possibly manually but it's not that big a deal, it's like 40 dashboards or so).

This solution is what we ended up doing for nearly all node exporter metrics in T213708. To this day, nearly every panel on the Host Overview dashboard has an or clause.

Yup, I must admit publicly that this trick was my inspiration for this proposal. But I had not thought of the or clause that's actually neat.

The proposed solution SGTM.

Many thanks!

Just chiming in to say that FWIW I concur with the or usage trick !

Using the python grafcli package I fetched all the dashboards under the Service/ folder, then sent them through the attached python script that did a number of regex substitutions.

The resulting JSONs I 've manually imported them checking dashboards. Most things are working just fine. There's a couple of issues here and there.

• In legends that used to be in the form blah {{ pod_name }} and now are blah {{ pod }} we no longer have the actual pod name in the old clusters. That's fine since we will soon be switching to the new kubernetes version the production clusters and that level of granularity will not be required for long.
• In the template variable container_name that is used in some dashboards. Rows using it (e.g. Saturation for $container_name row) it only works for the new scheme as I found no way to convince grafana (and it's (label_values() function) to populate data from 2 different labels (even for the same metric) nor have I found a way to concatenate them. Again, this should be fine, same reasoning as above.

Run the same thing on the Kubernetes/ folder. I only had to update 3 dashboards

• Kubernetes DNS
• Kubernetes Staging pods
• Kubernetes pods

All other dashboards were a no diff.