In T89222#1238317, @bd808 wrote:Ideally we would find a way to configure the application to ship logs directly to logstash and fluorine. Today we are shipping logs to fluorine via both our proprietary udp2log packet format and syslog. Logstash is actively accepting log events as GELF packets and syslog UDP datagrams. There are a number of additional log input methods that can be enabled for Logstash if needed.
If for some reason direct shipping is not possible, we should find a log shipper application that can tail the local disk logs and relay the events they represent to other hosts. If this is needed I'd like to work with someone from ops to find a single application that we can standardize on across production rather than having a one-off solution for each and every application that has an inflexible logging layer. There are a large number of solutions in this space including our own proprietary log2udp service, rsyslog's imfile feature, and dedicated log shipping services like beaver and lumberjack. I'd lean towards finding a dedicated shipper with an active community to standardize on as these tools generally provide a more flexible means to describe what constitutes a single log event than log2udp and rsyslog which are both strictly line oriented.
Description
Related Objects
- Mentioned In
- T157396: HHVM fills logstash with junk (due to not handling multiline errors?)
T109101: Currently elasticsearch logs do not leave nodes. We use logstash for this across the cluster generally.
T89222: export logs to logstash or create apertium-admins group on sca1001/sca1002 - Mentioned Here
- T89222: export logs to logstash or create apertium-admins group on sca1001/sca1002
Event Timeline
The only things I have personally used before to do this are the forwarder bundled with Splunk (proprietary) and local installs of Logstash (a bit RAM/CPU heavy due to use of JVM).
The forwarder formerly known as Lumberjack seems promising if only because it is maintained by Elastic and they also maintain Logstash. A possible negative however is that it is written in Go which brings a new language to the cluster if we find that we need to write a custom extension for some reason. On initial examination it looks like it would be non-trivial to extend it to also send log events to udp2log on fluorine.
We have the logging pipeline in production now, in other words applications send logs to either local syslog unix socket / journald or localhost udp