When I rebooted logstash1007-1009 for security updates, they had a failed service for prometheus-elasticsearch-exporter.service when they came back up:
sudo systemctl list-units | grep failed • prometheus-elasticsearch-exporter.service loaded failed failed Prometheus exporter for Elasticsearch
The actual exporter instance (-9200) seems to be running fine.
In journalctl it can be seen that this is caused by the missing Environment file:
Nov 28 10:49:52 logstash1009 systemd: Starting Prometheus exporter for Elasticsearch... Nov 28 10:49:52 logstash1009 systemd: Failed to load environment files: No such file or directory Nov 28 10:49:52 logstash1009 systemd: prometheus-elasticsearch-exporter.service failed to run 'start' task: No such file or directory
This environment file was explictly removed in puppet:
The same Puppet class also ensures that the exporter is stopped, but at that point it has already failed (it gets started upon system boot)
Given that the prometheus-elasticsearch-exporter ships a systemd unit we don't use (and which can't even start due to the removal of the environment file) it would be best if prometheus-elasticsearch-exporter were masked in puppet (via systemctl mask prometheus-elasticsearch-exporter.service).