Sounds good!

I spent some time yesterday looking into what this will take. It turns out to be not quite as easy as I'd hoped but it does seem possible. The Monolog SyslogUdpHandler is a bit janky. It creates an RFC 5424 syslog packet ... sort of. What it does is prepend a partial RFC 5424 header (PRI and VERSION) to whatever content is produced by the associated Formatter. The default formatter that is configured for use with this Handler does not seem to produce packets that rsyslog accepts as valid. The design also makes mixing SyslogUdpHandler with LogstashFormatter impossible. Through some trial and error I found that creating and sending RFC 3164 formatted messages is much easier and works both with rsyslog forwarding and direct Logstash syslog input.

Change 189672 had a related patch set uploaded (by BryanDavis):
Add Monolog handler for syslog UDP transport

https://gerrit.wikimedia.org/r/189672

Patch-For-Review

Change 189677 had a related patch set uploaded (by BryanDavis):
ELK: Log via syslog udp direct to Logstash

https://gerrit.wikimedia.org/r/189677

Patch-For-Review

The syslog transport will pass messages up to 65023 bytes minus the syslog message header without truncating. The message header will be 31 bytes + the length of php_uname( 'n' ) when using "mediawiki" as the app name. With the custom handler from https://gerrit.wikimedia.org/r/189672 can be used either with rsyslog as an intermediary or sent directly to Logstash as UDP packets.

If we wanted to use rsyslog as an intermediary, it would need configuration similar to this:

# Listen for UDP logging on 127.0.0.1:514
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

# Forward mediawiki events to logstash
if $programname == "mediawiki" then {
    @LOGSTASH_HOST:10514
    stop
}

If the aggregator was not per-host the listener would obviously need to be bound to an exposed ip on the host.

If we instead send the datagrams directly towards Logstash from MediaWiki no rsyslog changes would be needed. We already have Logstash listening for syslog traffic on port 10514. This is the transport currently used to collect Apache2 and HHVM log events.

In T88870#1025205, @bd808 wrote:

What it does is prepend a partial RFC 5424 header (PRI and VERSION) to whatever content is produced by the associated Formatter.

Looking at the version in mediawiki/vendor, it doesn't even seem to do that much. It has a PRI field, but then it outputs a colon where the VERSION should be.

In T88870#1028470, @Anomie wrote:

In T88870#1025205, @bd808 wrote:

What it does is prepend a partial RFC 5424 header (PRI and VERSION) to whatever content is produced by the associated Formatter.

Looking at the version in mediawiki/vendor, it doesn't even seem to do that much. It has a PRI field, but then it outputs a colon where the VERSION should be.

Oh wow. I think I was looking at a slightly newer version. That is completely broken.

Change 190231 had a related patch set uploaded (by BryanDavis):
logstash: Support MediaWiki logs via Syslog

https://gerrit.wikimedia.org/r/190231

Patch-For-Review

Change 190246 had a related patch set uploaded (by BryanDavis):
beta: switch logstash transport from redis to syslog

https://gerrit.wikimedia.org/r/190246

Patch-For-Review

Change 189672 merged by jenkins-bot:
Add Monolog handler for syslog UDP transport

https://gerrit.wikimedia.org/r/189672

Change 190246 merged by jenkins-bot:
beta: switch logstash transport from redis to syslog

https://gerrit.wikimedia.org/r/190246

Change 190629 had a related patch set uploaded (by BryanDavis):
Revert "Revert "beta: switch logstash transport from redis to syslog""

https://gerrit.wikimedia.org/r/190629

Patch-For-Review

Change 190629 merged by jenkins-bot:
Revert "Revert "beta: switch logstash transport from redis to syslog""

https://gerrit.wikimedia.org/r/190629

This code path is running in beta now and logs are showing up in Kibana as expected: https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/mediawiki

The full set of changes requires several changes that will be included in 1.25wmf18 so rollout to production will need at least that branch. Making backports of everything needed would be a large pain in the butt (monolog 1.12.0 in mediawiki/vendor, updates to MWLoggerMonologHandler, new MWLoggerMonologSyslogHandler handler).

Change 189677 merged by jenkins-bot:
ELK: Log via syslog udp direct to Logstash

https://gerrit.wikimedia.org/r/189677

Change 190231 merged by Gage:
logstash: Support MediaWiki logs via Syslog

https://gerrit.wikimedia.org/r/190231