In Jenkins service default file, we can set JENKINS_LOG to a syslog facility eg: daemon.info. The init script eventually pass it to the java daemon via --output=$JENKINS_LOG.

Updated the task detail:

• Jenkins is behind systemd now with logs relayed to syslog
• There is a logging property which could be tweaked to also relay log to logstash

Change 474990 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] jenkins: ship syslogs tagged 'jenkins' to ELK

https://gerrit.wikimedia.org/r/474990

Change 474990 merged by Filippo Giunchedi:
[operations/puppet@production] jenkins: ship syslogs tagged 'jenkins' to ELK

https://gerrit.wikimedia.org/r/474990

This is working, though we'll need to adjust the "level" field:

{
  "_index": "logstash-syslog-2018.11.21",
  "_type": "syslog",
  "_id": "AWc2ltxo8MXbBsCP-z72",
  "_version": 1,
  "_score": null,
  "_source": {
    "severity": "info",
    "level": "%{SEVERITY_LABEL}",
    "program": "jenkins",
    "message": "WARNING: [hudson.model.listeners.RunListener report] RunListener failed",
    "logsource": "contint1001",
    "type": "syslog",
    "normalized_message": "WARNING: [hudson.model.listeners.RunListener report] RunListener failed",
    "tags": [
      "rsyslog-shipper",
      "kafka",
      "syslog",
      "es",
      "normalized_message_untrimmed"
    ],
    "@timestamp": "2018-11-21T14:05:15.693Z",
    "host": "contint1001",
    "@version": "1",
    "facility": "daemon",
    "timestamp": "2018-11-21T14:05:14.469546+00:00"
  },
  "fields": {
    "@timestamp": [
      1542809115693
    ]
  },
  "sort": [
    1542809115693
  ]
}

Change 475104 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: rename 'severity' syslog field if present

https://gerrit.wikimedia.org/r/475104

Change 475110 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: add 'level' normalization rules

https://gerrit.wikimedia.org/r/475110

Jenkins emit the log to stdout or stderr which are captured by systemd and relayed to rsyslog (at least that is my understanding). But in the process the level/severity is lost, that forces us to have to process each lines on logstash side.

Gerrit is a java application as well, I am pretty sure @Paladox made it to send logs directly to rsyslog, thus with the proper syslog level/severity which is recognized by logstash generic rule.

I have updated the task description about Jenkins logging config and its configuration file. Probably we should align Jenkins with what has been done for Gerrit. It is probably better to have things correct at the source and eventually have less logic in logstash.

In T143733#4766671, @hashar wrote:

Jenkins emit the log to stdout or stderr which are captured by systemd and relayed to rsyslog (at least that is my understanding). But in the process the level/severity is lost, that forces us to have to process each lines on logstash side.

Gerrit is a java application as well, I am pretty sure @Paladox made it to send logs directly to rsyslog, thus with the proper syslog level/severity which is recognized by logstash generic rule.

I have updated the task description about Jenkins logging config and its configuration file. Probably we should align Jenkins with what has been done for Gerrit. It is probably better to have things correct at the source and eventually have less logic in logstash.

Agreed some context is lost via stdout/stderr and it'd be preferrable to use syslog directly instead. AFAIK Java can't use /dev/log out of the box, not sure about syslog() though, anyways something we should be doing for sure!

Change 475110 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: add 'level' normalization rules

https://gerrit.wikimedia.org/r/475110

Change 475104 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: rename 'severity' syslog field if present

https://gerrit.wikimedia.org/r/475104

In T143733#4767255, @fgiunchedi wrote:

Agreed some context is lost via stdout/stderr and it'd be preferrable to use syslog directly instead. AFAIK Java can't use /dev/log out of the box, not sure about syslog() though, anyways something we should be doing for sure!

I remember I talked with @Gehel about Java logging, he could be able to help polish up Jenkins logging.

I have dig a bit into it last night. Gerrit uses log4j which has support to relay to syslog. On the other hand, Jenkins uses the built-in Java mechanism java.util.logging which apparently does not come with a built-in handler for syslog :-(

In T143733#4767360, @hashar wrote:

In T143733#4767255, @fgiunchedi wrote:

Agreed some context is lost via stdout/stderr and it'd be preferrable to use syslog directly instead. AFAIK Java can't use /dev/log out of the box, not sure about syslog() though, anyways something we should be doing for sure!

I remember I talked with @Gehel about Java logging, he could be able to help polish up Jenkins logging.

I have dig a bit into it last night. Gerrit uses log4j which has support to relay to syslog. On the other hand, Jenkins uses the built-in Java mechanism java.util.logging which apparently does not come with a built-in handler for syslog :-(

For a simple approach we could use rsyslog::input::file to ingest the desired log file(s) with rsyslog for shipping.