We make use of "timers" with variables in scap, e.g. for scap deploy we show Finished deploy_foo/deploy (duration: 00m 29s)
These timers, optionally, send timing data to statsd.
This means graphite (and in future prometheus) has a lot of very long labels like:
- running_helm___kubeconfig__etc_kubernetes_mw_api_ext_deploy_codfw_config_rollback_canary_in__srv_deployment_charts_helmfile_d_services_mw_api_ext
- deploy__gerrit_gerrit_30691f2_
- scap_prep_1_43_0_wmf_9
We should normalize or eliminate timers containing variables to reduce cardinality in statsd and hopefully make our metrics more useful