Page MenuHomePhabricator

Improve HA for logstash cluster
Closed, ResolvedPublic


Before we can really start to rely on logstash it needs to have some work done to ensure that log events from the various input systems can reach the cluster via reliable transport and that various logstash nodes can consume that input.

In the current udp2log relay setup we are really only using the logstash1001 instance to process all incoming logs. Any time this node is restarted all log events are lost until it comes back up (2-3 minutes).

Version: wmf-deployment
Severity: major



Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:03 AM
bzimport set Reference to bz61785.
bzimport added a subscriber: Unknown Object (MLST).

Why not use Kafka as the messaging bus? That would solve all your reliability / durability concerns, it's operated by Ops for the Analytics team so it builds on existing infrastructure and there seems to be producer/consumer for logstash available at (hahaha more debianization fun)

Hadoop uses Logstash-gelf.jar ( which supports Redis but not Kafka. Though maybe we could submit a patch...

Just as a historical point of interest related to this feature request, we briefly switched MediaWiki logging to ship to Logstash via Redis and found that it was slowing things down measurably from the MediaWiki side.

UDP seems to be an ok transport for debug log messages, but it would be nice to have a more robust way to distribute the traffic across multiple Logstash backends. MediaWiki has switched from udp2log to using syslog udp datagrams as the transport mechanism. It is also configured to randomly select one of the Logstash servers for all log events from a given request.

fgiunchedi claimed this task.