Page MenuHomePhabricator

Send Striker logs to Logstash
Closed, ResolvedPublic

Description

Figure out the right configuration to send the uwsgi and django log streams from Striker on californium to Logstash.

Some things to look at:

This might also be a good time to make the logging configuration layer in Striker easier to override via Puppet generated files.

Event Timeline

bd808 created this task.Aug 16 2016, 10:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 16 2016, 10:59 PM

Change 305941 had a related patch set uploaded (by BryanDavis):
Add Logstash logging support

https://gerrit.wikimedia.org/r/305941

Change 306047 had a related patch set uploaded (by BryanDavis):
Add python-logstash

https://gerrit.wikimedia.org/r/306047

Change 306049 had a related patch set uploaded (by BryanDavis):
Add python-logstash and bump wheels

https://gerrit.wikimedia.org/r/306049

Change 306055 had a related patch set uploaded (by BryanDavis):
logstash: Tag Striker messages for indexing

https://gerrit.wikimedia.org/r/306055

Change 305941 merged by jenkins-bot:
Add Logstash logging support

https://gerrit.wikimedia.org/r/305941

Change 306047 merged by jenkins-bot:
Add python-logstash

https://gerrit.wikimedia.org/r/306047

Change 306049 merged by jenkins-bot:
Add logstash logging support

https://gerrit.wikimedia.org/r/306049

bd808 updated the task description. (Show Details)Aug 22 2016, 11:41 PM

Change 306081 had a related patch set uploaded (by BryanDavis):
logstash: new input for msgpack over UDP

https://gerrit.wikimedia.org/r/306081

Change 306081 merged by Gehel:
logstash: new input for msgpack over UDP

https://gerrit.wikimedia.org/r/306081

The uwsgi side ended up being a bit uglier than I had hoped, but I got it to work on my testing cluster in Labs. The working uwsgi logging config looks something like this:

[uwsgi]
# Keep a copy of all stout/stderr logs locally
logger=local file:/srv/log/striker/main.log
log-route=local .*
log-encoder=format:local [${strftime:%%Y-%%m-%%dT%%H:%%M:%%S}] ${msgnl}

# Send a copy of all stdout/stderr logs to logstash as json datagrams
logger=logstash socket:deployment-logstash2.eqiad.wmflabs:11514
log-route=logstash .*
log-encoder=json:logstash {"@timestamp":"${strftime:%%Y-%%m-%%dT%%H:%%M:%%S}","type":"striker","logger_name":"uwsgi","host":"%h","level":"INFO","message":"${msg}"}

# Keep WMF-style Apache combined access logs locally
req-logger=file:/srv/log/striker/access.log
log-format=%(addr) - %(user) [%(ltime)] "%(method) %(uri) (proto)" %(status) %(size) "%(referer)" "%(uagent)" %(micros)

I theory the log-route settings are not needed, but in practice I couldn't get everything to work as desired without them.

Change 306055 merged by Yuvipanda:
logstash: Tag Striker messages for indexing

https://gerrit.wikimedia.org/r/306055

bd808 moved this task from Backlog to Doing on the Striker board.Aug 25 2016, 9:39 PM
Restricted Application added a project: User-bd808. · View Herald TranscriptAug 25 2016, 10:24 PM
bd808 moved this task from To Do to Archive on the User-bd808 board.Sep 1 2016, 5:14 PM
bd808 moved this task from Doing to Done on the Striker board.Mar 4 2017, 3:57 AM