Page MenuHomePhabricator

Send Jenkins daemon logs to logstash
Open, MediumPublic

Description

As we did / are doing for Gerrit with T141324 we could use the Jenkins daemon logs (not the build console / result) to be sent to logstash.

@demon pointed at https://blog.lanyonm.org/articles/2015/12/29/log-aggregation-log4j-spring-logstash.html#log4j-over-tcp

Need a log4j.property file and figure out how much log crazy Jenkins might end up sending.


Migrated Jenkins to use systemd via Puppet patch 07c8bb1ab3 and others. The logging is emitted to stdout, captured by systemd and end up in syslog. We lost the log level in the process though so that is not ideal.

42d145823f introduces a property file to configure logging, that can probably be tweaked to have Jenkins to additionally send logs to logstash?

modules/jenkins/templates/logging.properties.erb
handlers=java.util.logging.ConsoleHandler

# https://docs.oracle.com/javase/7/docs/api/java/util/Formatter.html
#
# Time is inserted by syslog. Else one could use: %1$tF %1$tT.%1$tL
#
# Arguments from Jenkins:
#  - %2 is the class/method
#  - %4 log level
#  - %5 message
java.util.logging.SimpleFormatter.format = %4$s: [%2$s] %5$s%n

java.util.logging.ConsoleHandler.level = WARNING
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

For Gerrit

It is invoked with java -Dlog4j.configuration=file:///var/lib/gerrit2/review_site/etc/log4j.xml which has all the logging configuration. This way the service owner can tweak which logs are emitted rather "easily".

Event Timeline

hashar created this task.Aug 23 2016, 7:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 23 2016, 7:47 PM
hashar triaged this task as Medium priority.Aug 23 2016, 8:04 PM
Paladox added a subscriber: Paladox.

In Jenkins service default file, we can set JENKINS_LOG to a syslog facility eg: daemon.info. The init script eventually pass it to the java daemon via --output=$JENKINS_LOG.

hashar updated the task description. (Show Details)May 3 2017, 2:17 PM

Updated the task detail:

  • Jenkins is behind systemd now with logs relayed to syslog
  • There is a logging property which could be tweaked to also relay log to logstash

Change 474990 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] jenkins: ship syslogs tagged 'jenkins' to ELK

https://gerrit.wikimedia.org/r/474990

Change 474990 merged by Filippo Giunchedi:
[operations/puppet@production] jenkins: ship syslogs tagged 'jenkins' to ELK

https://gerrit.wikimedia.org/r/474990

This is working, though we'll need to adjust the "level" field:

{
  "_index": "logstash-syslog-2018.11.21",
  "_type": "syslog",
  "_id": "AWc2ltxo8MXbBsCP-z72",
  "_version": 1,
  "_score": null,
  "_source": {
    "severity": "info",
    "level": "%{SEVERITY_LABEL}",
    "program": "jenkins",
    "message": "WARNING: [hudson.model.listeners.RunListener report] RunListener failed",
    "logsource": "contint1001",
    "type": "syslog",
    "normalized_message": "WARNING: [hudson.model.listeners.RunListener report] RunListener failed",
    "tags": [
      "rsyslog-shipper",
      "kafka",
      "syslog",
      "es",
      "normalized_message_untrimmed"
    ],
    "@timestamp": "2018-11-21T14:05:15.693Z",
    "host": "contint1001",
    "@version": "1",
    "facility": "daemon",
    "timestamp": "2018-11-21T14:05:14.469546+00:00"
  },
  "fields": {
    "@timestamp": [
      1542809115693
    ]
  },
  "sort": [
    1542809115693
  ]
}

Change 475104 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: rename 'severity' syslog field if present

https://gerrit.wikimedia.org/r/475104

Change 475110 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: add 'level' normalization rules

https://gerrit.wikimedia.org/r/475110

Jenkins emit the log to stdout or stderr which are captured by systemd and relayed to rsyslog (at least that is my understanding). But in the process the level/severity is lost, that forces us to have to process each lines on logstash side.

Gerrit is a java application as well, I am pretty sure @Paladox made it to send logs directly to rsyslog, thus with the proper syslog level/severity which is recognized by logstash generic rule.

I have updated the task description about Jenkins logging config and its configuration file. Probably we should align Jenkins with what has been done for Gerrit. It is probably better to have things correct at the source and eventually have less logic in logstash.

hashar updated the task description. (Show Details)Nov 21 2018, 8:18 PM
hashar updated the task description. (Show Details)

Jenkins emit the log to stdout or stderr which are captured by systemd and relayed to rsyslog (at least that is my understanding). But in the process the level/severity is lost, that forces us to have to process each lines on logstash side.

Gerrit is a java application as well, I am pretty sure @Paladox made it to send logs directly to rsyslog, thus with the proper syslog level/severity which is recognized by logstash generic rule.

I have updated the task description about Jenkins logging config and its configuration file. Probably we should align Jenkins with what has been done for Gerrit. It is probably better to have things correct at the source and eventually have less logic in logstash.

Agreed some context is lost via stdout/stderr and it'd be preferrable to use syslog directly instead. AFAIK Java can't use /dev/log out of the box, not sure about syslog() though, anyways something we should be doing for sure!

Change 475110 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: add 'level' normalization rules

https://gerrit.wikimedia.org/r/475110

Change 475104 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: rename 'severity' syslog field if present

https://gerrit.wikimedia.org/r/475104

hashar added a subscriber: Gehel.Nov 22 2018, 9:26 AM

Agreed some context is lost via stdout/stderr and it'd be preferrable to use syslog directly instead. AFAIK Java can't use /dev/log out of the box, not sure about syslog() though, anyways something we should be doing for sure!

I remember I talked with @Gehel about Java logging, he could be able to help polish up Jenkins logging.

I have dig a bit into it last night. Gerrit uses log4j which has support to relay to syslog. On the other hand, Jenkins uses the built-in Java mechanism java.util.logging which apparently does not come with a built-in handler for syslog :-(

This comment was removed by hashar.
herron added a subscriber: herron.Nov 26 2018, 7:12 PM

Agreed some context is lost via stdout/stderr and it'd be preferrable to use syslog directly instead. AFAIK Java can't use /dev/log out of the box, not sure about syslog() though, anyways something we should be doing for sure!

I remember I talked with @Gehel about Java logging, he could be able to help polish up Jenkins logging.

I have dig a bit into it last night. Gerrit uses log4j which has support to relay to syslog. On the other hand, Jenkins uses the built-in Java mechanism java.util.logging which apparently does not come with a built-in handler for syslog :-(

For a simple approach we could use rsyslog::input::file to ingest the desired log file(s) with rsyslog for shipping.

fgiunchedi moved this task from Inbox to Radar on the observability board.Mon, Jul 20, 1:29 PM