Page MenuHomePhabricator

prometheus: figure out how to clean metrics for decommissioned hardware
Closed, DuplicatePublic

Description

In this dashboard https://grafana.wikimedia.org/d/000000579/wmcs-openstack-eqiad-summary?orgId=1 we have metrics for hosts that no longer exist.

Figure out how to clean those.

Event Timeline

Prometheus configs are generally driven by Puppet (e.g. via exported resources or hiera lists), in other words hosts should disappear as soon as they are deactivated in Puppet. What hosts are decom that still show up in the dashboard ?

these here:

image.png (523×1 px, 168 KB)

Now that I realize it, node metrics aren't present, but those from the openstack exporter.

This task is actually a duplicate of T335978: openstack: consider removing references to old hardware from the database