Page MenuHomePhabricator

Improve HA for logstash cluster
Closed, ResolvedPublic

Description

Before we can really start to rely on logstash it needs to have some work done to ensure that log events from the various input systems can reach the cluster via reliable transport and that various logstash nodes can consume that input.

In the current udp2log relay setup we are really only using the logstash1001 instance to process all incoming logs. Any time this node is restarted all log events are lost until it comes back up (2-3 minutes).


Version: wmf-deployment
Severity: major

Details

Reference
bz61785

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:03 AM
bzimport set Reference to bz61785.
bzimport added a subscriber: Unknown Object (MLST).
bd808 created this task.Feb 21 2014, 11:00 PM

<cool_aid_advertisement>
Why not use Kafka as the messaging bus? That would solve all your reliability / durability concerns, it's operated by Ops for the Analytics team so it builds on existing infrastructure and there seems to be producer/consumer for logstash available at https://github.com/joekiller/logstash-kafka (hahaha more debianization fun)
</cool_aid_advertisement>

Gage merged a task: Restricted Task.Dec 18 2014, 6:25 PM
Gage added subscribers: Aklapper, demon, Gage, GWicke.

Hadoop uses Logstash-gelf.jar (https://github.com/mp911de/logstash-gelf) which supports Redis but not Kafka. Though maybe we could submit a patch...

bd808 added a comment.Jul 27 2015, 8:27 PM

Just as a historical point of interest related to this feature request, we briefly switched MediaWiki logging to ship to Logstash via Redis and found that it was slowing things down measurably from the MediaWiki side.

UDP seems to be an ok transport for debug log messages, but it would be nice to have a more robust way to distribute the traffic across multiple Logstash backends. MediaWiki has switched from udp2log to using syslog udp datagrams as the transport mechanism. It is also configured to randomly select one of the Logstash servers for all log events from a given request.

fgiunchedi moved this task from Backlog to Up next on the Wikimedia-Logstash board.Aug 6 2018, 1:13 PM
fgiunchedi closed this task as Resolved.Oct 1 2018, 1:12 PM
fgiunchedi claimed this task.