Target definitions as expanded by puppet are identical on both servers too. Very weird.

tcpdump & netstat confirmed that prometheus2004 wasn't even trying to connect to lvs2007|8:9100, a restart has fixed it.

Oh, also NB that a simple reload was not sufficient to fix (tried that first).

Curious indeed, thanks for investigating! I took a quick look and it looks like prometheus2004 didn't even know about lvs2007 before today at ~6.23, so I'm suspecting watching targets files failed in some way / for some reason. I can't find any smoking gun in prometheus metrics ATM we could check for alerting, although we should be updating to newer prometheus releases (e.g. 2.15)

I am investigating a similar problem for some new elasticsearch nodes (like elastic2055). If I create a ssh tunnel for prometheus2003 and 2004, I can see the metrics popping up in the web UI of 2004 but not on 2003.

The issue went away by itself after mentioning it to Chris and Filippo :)

I took a look at this on both prometheus200[34] for up{instance=~"elastic2055.*9108"} and the metric appears yesterday on 2004 at 9:44 and 2003 at 17:48. Whereas both target files (/srv/prometheus/ops/targets/elasticsearch_codfw.yaml) are last modified at 10:10, so definitely 2003 has been lagging behind and eventually discovered the target, for reasons yet to be determined.

There was a reload of prometheus@ops on 2003 at 17:41.

I think the following shell one-liner could be turned into an Icinga check for this condition pretty easily:

25h later, there's additional skew between 1003 and 1004:

2003/2004 are both presently identical.

As of this morning: 2003/2004 presently identical.

Skew still increased on 1003/1004:

@Aklapper: "After extensive research, it has been determined that the correct plural of 'Prometheus' is 'Prometheis'."
— https://prometheus.io/docs/introduction/faq/#what-is-the-plural-of-prometheus

It appears a reload does resolve the issue, but it takes some time for Prometheus to fetch and store an update. I used kill -HUP <PID> to reload.

The logs immediately following a reload seem to indicate the process cannot watch the config files for changes. Puppet does not issue a reload when target files are modified in any way.

# Issued HUP here
Mar 13 00:20:34 prometheus1003 prometheus@ops[20969]: level=info ts=2020-03-13T00:20:34.167231405Z caller=main.go:698 msg="Loading configuration file" filename=/srv/prometheus/ops/prometheus.yml
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=info ts=2020-03-13T00:20:38.818681106Z caller=main.go:725 msg="Completed loading of configuration file" filename=/srv/prometheus/ops/prometheus.yml
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.852335966Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.860305321Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.860310584Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.860314889Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.860316497Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.860315009Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.860302454Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.860310984Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.860309546Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"
Mar 13 00:20:38 prometheus1003 prometheus@ops[20969]: level=error ts=2020-03-13T00:20:38.860309539Z caller=file.go:223 component="discovery manager scrape" discovery=file msg="Error adding file watcher" err="too many open files"

Nice catch in the logs!

I'm guessing we need to increase some of the inotify tunables. Probably this one:

✔️ cdanis@prometheus1003.eqiad.wmnet ~ 🕣🍺 cat /proc/sys/fs/inotify/max_user_watches
8192

There's only so many files to be watched, but judging from lsof output, each prometheus thread holds an inotify handle on all of them, so it adds up quickly.

Less likely (but possible) is needing to also increase fs.inotify.max_user_instances = 128

Change 579453 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] prom@ops: increase inotify watches in core DCs

https://gerrit.wikimedia.org/r/579453

Change 579453 merged by Cwhite:
[operations/puppet@production] prom@ops: increase inotify watches in core DCs

https://gerrit.wikimedia.org/r/579453

Change 579613 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: increase inotify instances in core DCs

https://gerrit.wikimedia.org/r/579613

Change 579613 merged by Cwhite:
[operations/puppet@production] profile: increase inotify instances in core DCs

https://gerrit.wikimedia.org/r/579613

Found a related issue.

We've upped max_user_instances and max_user_watches and the error logs haven't come back.

Let's check back up on this in a week or so.

@Krinkle please see https://phabricator.wikimedia.org/T246860#5964089 :)

Thanks... I guess that's bound to cause confusion, but I will try to be one of the people that does know and remembers.