Page MenuHomePhabricator
Create Task
Maniphest T141783

Add monitoring for detecting when logstash services are down
Closed, InvalidPublic
Actions

Description

See T141776: Parsoid's service-runner events not showing up in Kibana since 2016-07-28 21:00 UTC. I first noticed this Saturday (30th July noon CT), but thought it was only Parsoid.

Looks like some monitoring of logstash service would be useful to ensure that we catch logstash service downtimes.

Related Objects

Event Timeline

ssastry created this task.Aug 1 2016, 4:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 1 2016, 4:16 PM
ssastry renamed this task from Add monitoring for ensuring logstash services are operational to Add monitoring for detecting when logstash services are down.Aug 1 2016, 4:17 PM
bd808 mentioned this in T141784: Log event rate types from Logstash to graphite.Aug 1 2016, 4:41 PM
bd808 subscribed.Aug 1 2016, 4:59 PM

We have monitoring for the service proper, but what has seemed to happen several times is that the java process gets hung up in a gc cycle and stops processing events.

akosiaris triaged this task as Medium priority.Aug 11 2016, 2:33 PM
fgiunchedi closed this task as Invalid.Oct 15 2018, 2:27 PM
fgiunchedi subscribed.

I don't think we've seen reoccurrence of this, also logstash now has monitoring for udp packet loss which I'm assuming would also show up if logstash services are down.

fgiunchedi added a project: observability.Aug 19 2019, 2:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL