Yes, I think we should look into sending cassandra logs to logstash. That way it should be easier to plot trends, set up alerts etc.

Since Cassandra uses the /log4j/ back-end for logging, this should be straightforward to configure using Logstash' log4j receiver.

In T100970#1328368, @mobrovac wrote:

Since Cassandra uses the /log4j/ back-end for logging, this should be straightforward to configure using Logstash' log4j receiver.

Cassandra actually uses logback these days, I've been eying this: https://github.com/logstash/logstash-logback-encoder

I know there was some concern about the appender blocking in the event it can not write to logstash, according to the README for logstassh-logback-encoder, this shouldn't be a problem:

Internally, the TCP appenders are asynchronous (using the LMAX Disruptor RingBuffer). All the encoding and TCP communication is delegated to a single writer thread. There is no need to wrap the TCP appenders with another asynchronous appender (such as AsyncAppender or LoggingEventAsyncDisruptorAppender).

All the configuration parameters (except for sub-appender) of the async appenders are valid for TCP appenders. For example, waitStrategyType and ringBufferSize.

The TCP appenders will never block the logging thread. If the RingBuffer is full (e.g. due to slow network, etc), then events will be dropped.

I'll do some testing of this and report back with my findings.

In T100970#1341509, @Eevans wrote:

Cassandra actually uses logback these days, I've been eying this: https://github.com/logstash/logstash-logback-encoder

Looks good (on paper)! :)

The TCP appenders will never block the logging thread. If the RingBuffer is full (e.g. due to slow network, etc), then events will be dropped.

There's also a UDP adapter, which I'd honestly prefer over the TCP one in spite of the ring buffer. UDP is async and non-blocking by design, and I think we all know how Java and TCP sockets work great together until they don't, which also seems to be true in this case.

That said, I think we should evaluate both options. Specifically, testing with looong messages containing tons of back-trace lines should be used.

In T100970#1341600, @mobrovac wrote:

There's also a UDP adapter, which I'd honestly prefer over the TCP one in spite of the ring buffer. UDP is async and non-blocking by design, and I think we all know how Java and TCP sockets work great together until they don't, which also seems to be true in this case.

If that's true, that networking in Java is somehow inherently unreliable, then I fear we have problems much larger than this. :)

The only reason that I've leaned toward TCP here, is that our Cassandra nodes are quite a bit chattier than say RB; From the looks of it we're doing something like 32 messages / second. I don't expect that to be a problem all by itself, but if we use UDP everywhere, then its not hard to imagine it causing some of the same problems that it currently does for Graphite (https://phabricator.wikimedia.org/T101141).

That said, I think we should evaluate both options. Specifically, testing with looong messages containing tons of back-trace lines should be used.

I agree.

To test, I looped indefinitely, logging messages ranging in size from 1 to 1000 words. The code I used is attached, (build w/ mvn package, and run w/ mvn exec:java -Dexec.mainClass=org.wikimedia.logstash.Logstuff).

For both appenders (UDP and TCP), I started and stopped logstash for varying periods of time (upwards of 30 minutes), periodically blocked network access entirely with iptables (DROP and REJECT), and simulated lossy connections (e.g. w/ iptables -A INPUT -i lo -m statistic --mode random --probability 0.1 -p tcp --dport 4050 -j DROP). Both appenders failed gracefully without blocking the application, and both resumed logging immediately as soon as connectivity was restored.

Change 219210 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: ship logback.xml with longer retention periods

https://gerrit.wikimedia.org/r/219210

Change 219210 merged by Filippo Giunchedi:
cassandra: ship logback.xml with longer retention periods

https://gerrit.wikimedia.org/r/219210

The only reason that I've leaned toward TCP here, is that our Cassandra nodes are quite a bit chattier than say RB; From the looks of it we're doing something like 32 messages / second.

That's actually a couple of orders of magnitude less chatty than the RB -> graphite traffic.

Change 220764 had a related patch set uploaded (by Eevans):
logstash-logback-encoder setup

https://gerrit.wikimedia.org/r/220764

Change 220764 merged by Eevans:
logstash-logback-encoder setup

https://gerrit.wikimedia.org/r/220764

Change 226025 had a related patch set uploaded (by Eevans):
WIP: Cassanra logstash setup

https://gerrit.wikimedia.org/r/226025

Change 226025 merged by Filippo Giunchedi:
Cassandra logstash setup

https://gerrit.wikimedia.org/r/226025