To some extend related: T109279: Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections.

gerrit process monitoring:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=ytterbium&service=gerrit+process

phabricator http monitoring:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=iridium&service=https%3A%2F%2Fphabricator.wikimedia.org

Gerrit availabilty:

http://status.wikimedia.org/8777/249692/Gerrit

Phabricator availability:

http://status.wikimedia.org/8777/388149/Phabricator

probably additionally Catchpoint too in addition to Icinga and Watchmouse.

Just to be sure: we are NOT talking about the traditional sense of HA (https://en.wikipedia.org/wiki/High_availability) with things like redundancy, quick failover, etc. That'd be way out of the ordinary for our dev tools (not that it wouldn't be good in theory, but way more effort than WMF has ever given).

In T115611#1730626, @Dzahn wrote:

Gerrit availabilty:

http://status.wikimedia.org/8777/249692/Gerrit

Phabricator availability:

http://status.wikimedia.org/8777/388149/Phabricator

probably additionally Catchpoint too in addition to Icinga and Watchmouse.

Are these enough? They seem like they give us "monitoring of errors and latency".

In T115611#1730798, @greg wrote:

Are these enough? They seem like they give us "monitoring of errors and latency".

I don't know much about the current monitoring stack, but my main question would be: do failures get reported to IRC? That's all that really matters, I imagine. (Though I personally found that awful icinga-wm bot annoying enough to ignore it. Presumably other people are listening to it.)

Yes, icinga announces in IRC when either of those two things listed above (the incinga links) fails.

icinga has paged me, and opsen, on multiple occasions when phabricator was down. I'm pretty sure that it's working.