Page MenuHomePhabricator

Stream Thumbor logs to logstash
Open, LowPublic

Description

Currently Thumbor logs can be found either via systemd's journal or under /srv/log/thumbor. We should figure out how to stream logs to logstash and view them in kibana.

Event Timeline

jijiki created this task.Jan 4 2019, 3:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2019, 3:45 PM
jijiki updated the task description. (Show Details)Jan 4 2019, 3:45 PM
herron triaged this task as Normal priority.Jan 4 2019, 5:22 PM
herron added a project: Wikimedia-Logstash.
herron added a subscriber: herron.

Sure! In terms of figuring out how, since the logs are already present in syslog this should be a pretty straightforward case.

We can approach it by first including ::profile::rsyslog::kafka_shipper in the appropriate thumbor role. This will plumb the hosts with rsyslog kafka dependencies and the rsyslog lookup/output configs for shipping to logstash.

Then, we can update the rsyslog lookup table at modules/profile/files/rsyslog/lookup_table_output.json to direct syslogs with thumbor program names to both kafka (logstash) and local outputs with config lines to the effect of:

{"index" : "thumbor@8801", "value" : "kafka local" },

Since thumbor hosts have something like 40 thumbor instances, each with unique program names, it will need 40 lines in this file. This isn't ideal, but afaik lookup tables are not currently able to match by "startswith" in the same way that the local rsyslog omfile rule is configured. Ran into a similar case with prometheus. Maybe there's a way to work around it, but listing them down would be good enough for now.

@jijiki did you want to take a stab at a patch for this? If not no worries, I'd be happy to do it.

Also all but the chattier thumbor logs are already sent to logstash (cfr T150734), this will it'll be a good occasion to move thumbor to the new logging infrastructure as @herron mentioned though!

jijiki added a comment.EditedJan 8 2019, 10:46 AM

@fgiunchedi @herron I agree, it is a good opportunity to move it to the new infrastructure:D. Since we will be upgrading Thumbor servers to Stretch soon, we could first do the upgrade along with T212941 and then go ahead and do T212946 and T187765

@herron should I ping you when we get to the logging part? Depending on availability and needs, we can figure out how we will implement it

Sure, sounds good!

jijiki lowered the priority of this task from Normal to Low.Jun 21 2019, 12:05 PM