Perhaps i'm just not finding them, but it doesn't look like cloudelastic metrics are making it to prometheus. Our dashboards, even the generic ones like Cluster Overview and Host overview do not list the cloudelastic servers.
A quick check of things I could think of that have to be in place:
- prometheus-node-exporter is running and listening on :9100
- ferm looks to have appropriate rules allowing connection from prometheus machines:
- &R_SERVICE(tcp, 9100, (@resolve((prometheus1003.eqiad.wmnet prometheus1004.eqiad.wmnet)) @resolve((prometheus1003.eqiad.wmnet prometheus1004.eqiad.wmnet), AAAA)));
- fetching localhost:9100/metrics returns what looks to be appropriate metrics
Random guess at problems:
- Could the *.wikimedia.org domain be throwing things off? Maybe eqiad prometheus only wants to talk to *.eqiad.wmnet?
Might require someone that can login to the prometheus servers and double check whats going on with it's configuration.