While investigating puppet failures in T299468 I noticed statsd-exporter couldn't parse its config anymore, and indeed it looks like certain yaml fields went from float to string:
Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) +--- Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) defaults: Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) - quantiles: Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) - - error: 0.001 Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) - quantile: 0.99 Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) - - error: 0.001 Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) - quantile: 0.95 Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) - - error: 0.001 Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) - quantile: 0.75 Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) - - error: 0.005 Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) - quantile: 0.5 Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) timer_type: summary Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) + quantiles: Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) + - quantile: '0.99' Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) + error: '0.001' Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) + - quantile: '0.95' Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) + error: '0.001' Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) + - quantile: '0.75' Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) + error: '0.001' Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) + - quantile: '0.50' Feb 17 20:03:32 ms-be2050 p...5]: (/...e[/etc/prometheus/statsd_exporter.conf]/content) + error: '0.005'
The fields are strings in puppet so it makes sense to write the values quoted in the file. This change is due to the move from ordered_yaml to to_yaml, I'll send a patch to fix the values in puppet. cc @jhathaway as there might be other cases lurking around of yaml configs (previously) working by chance).
This problem is an unfortunate combination of running an old version of statsd-exporter that doesn't support checking its config (and thus allow us to use validate_cmd in puppet)