Page MenuHomePhabricator

Prototype Monolog and rsyslog configuration to ship log events from MediaWiki to Logstash
Closed, ResolvedPublic

Description

Setup a test environment and configure Monolog to send log events to an rsyslog service running on the same host and then use rsyslog forwarding to bounce those events to Logstash. Test with both UDP and TCP syslog connections and determine maximum message length allowed before rsyslog truncates the payload.

Event Timeline

bd808 claimed this task.
bd808 raised the priority of this task from to High.
bd808 updated the task description. (Show Details)
bd808 moved this task from Backlog to In Dev/Progress on the MediaWiki-Core-Team board.

Sounds good!

I spent some time yesterday looking into what this will take. It turns out to be not quite as easy as I'd hoped but it does seem possible. The Monolog SyslogUdpHandler is a bit janky. It creates an RFC 5424 syslog packet ... sort of. What it does is prepend a partial RFC 5424 header (PRI and VERSION) to whatever content is produced by the associated Formatter. The default formatter that is configured for use with this Handler does not seem to produce packets that rsyslog accepts as valid. The design also makes mixing SyslogUdpHandler with LogstashFormatter impossible. Through some trial and error I found that creating and sending RFC 3164 formatted messages is much easier and works both with rsyslog forwarding and direct Logstash syslog input.

gerritbot added a subscriber: gerritbot.

Change 189672 had a related patch set uploaded (by BryanDavis):
Add Monolog handler for syslog UDP transport

https://gerrit.wikimedia.org/r/189672

Patch-For-Review

Change 189677 had a related patch set uploaded (by BryanDavis):
ELK: Log via syslog udp direct to Logstash

https://gerrit.wikimedia.org/r/189677

Patch-For-Review

The syslog transport will pass messages up to 65023 bytes minus the syslog message header without truncating. The message header will be 31 bytes + the length of php_uname( 'n' ) when using "mediawiki" as the app name. With the custom handler from https://gerrit.wikimedia.org/r/189672 can be used either with rsyslog as an intermediary or sent directly to Logstash as UDP packets.

If we wanted to use rsyslog as an intermediary, it would need configuration similar to this:

# Listen for UDP logging on 127.0.0.1:514
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

# Forward mediawiki events to logstash
if $programname == "mediawiki" then {
    @LOGSTASH_HOST:10514
    stop
}

If the aggregator was not per-host the listener would obviously need to be bound to an exposed ip on the host.

If we instead send the datagrams directly towards Logstash from MediaWiki no rsyslog changes would be needed. We already have Logstash listening for syslog traffic on port 10514. This is the transport currently used to collect Apache2 and HHVM log events.

Anomie added a subscriber: Anomie.Feb 10 2015, 5:49 PM

What it does is prepend a partial RFC 5424 header (PRI and VERSION) to whatever content is produced by the associated Formatter.

Looking at the version in mediawiki/vendor, it doesn't even seem to do that much. It has a PRI field, but then it outputs a colon where the VERSION should be.

What it does is prepend a partial RFC 5424 header (PRI and VERSION) to whatever content is produced by the associated Formatter.

Looking at the version in mediawiki/vendor, it doesn't even seem to do that much. It has a PRI field, but then it outputs a colon where the VERSION should be.

Oh wow. I think I was looking at a slightly newer version. That is completely broken.

Change 190231 had a related patch set uploaded (by BryanDavis):
logstash: Support MediaWiki logs via Syslog

https://gerrit.wikimedia.org/r/190231

Patch-For-Review

Change 190246 had a related patch set uploaded (by BryanDavis):
beta: switch logstash transport from redis to syslog

https://gerrit.wikimedia.org/r/190246

Patch-For-Review

Change 189672 merged by jenkins-bot:
Add Monolog handler for syslog UDP transport

https://gerrit.wikimedia.org/r/189672

Change 190246 merged by jenkins-bot:
beta: switch logstash transport from redis to syslog

https://gerrit.wikimedia.org/r/190246

Change 190629 had a related patch set uploaded (by BryanDavis):
Revert "Revert "beta: switch logstash transport from redis to syslog""

https://gerrit.wikimedia.org/r/190629

Patch-For-Review

Change 190629 merged by jenkins-bot:
Revert "Revert "beta: switch logstash transport from redis to syslog""

https://gerrit.wikimedia.org/r/190629

This code path is running in beta now and logs are showing up in Kibana as expected: https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/mediawiki

The full set of changes requires several changes that will be included in 1.25wmf18 so rollout to production will need at least that branch. Making backports of everything needed would be a large pain in the butt (monolog 1.12.0 in mediawiki/vendor, updates to MWLoggerMonologHandler, new MWLoggerMonologSyslogHandler handler).

Change 189677 merged by jenkins-bot:
ELK: Log via syslog udp direct to Logstash

https://gerrit.wikimedia.org/r/189677

Change 190231 merged by Gage:
logstash: Support MediaWiki logs via Syslog

https://gerrit.wikimedia.org/r/190231

bd808 closed this task as Resolved.Feb 25 2015, 5:35 PM
bd808 moved this task from Done to Archive on the MediaWiki-Core-Team board.Mar 3 2015, 5:00 PM
bd808 moved this task from Backlog to Archive on the Wikimedia-Logstash board.Apr 22 2015, 3:25 AM
Restricted Application added a project: User-bd808. · View Herald TranscriptJul 29 2016, 8:19 PM
bd808 moved this task from To Do to Archive on the User-bd808 board.Sep 1 2016, 5:14 PM