Page MenuHomePhabricator

Stream Thumbor logs to logstash
Closed, ResolvedPublic


Currently Thumbor logs can be found either via systemd's journal or under /srv/log/thumbor. We should figure out how to stream logs to logstash and view them in kibana.

Event Timeline

herron triaged this task as Medium priority.Jan 4 2019, 5:22 PM
herron added a project: Wikimedia-Logstash.
herron subscribed.

Sure! In terms of figuring out how, since the logs are already present in syslog this should be a pretty straightforward case.

We can approach it by first including ::profile::rsyslog::kafka_shipper in the appropriate thumbor role. This will plumb the hosts with rsyslog kafka dependencies and the rsyslog lookup/output configs for shipping to logstash.

Then, we can update the rsyslog lookup table at modules/profile/files/rsyslog/lookup_table_output.json to direct syslogs with thumbor program names to both kafka (logstash) and local outputs with config lines to the effect of:

{"index" : "thumbor@8801", "value" : "kafka local" },

Since thumbor hosts have something like 40 thumbor instances, each with unique program names, it will need 40 lines in this file. This isn't ideal, but afaik lookup tables are not currently able to match by "startswith" in the same way that the local rsyslog omfile rule is configured. Ran into a similar case with prometheus. Maybe there's a way to work around it, but listing them down would be good enough for now.

@jijiki did you want to take a stab at a patch for this? If not no worries, I'd be happy to do it.

Also all but the chattier thumbor logs are already sent to logstash (cfr T150734), this will it'll be a good occasion to move thumbor to the new logging infrastructure as @herron mentioned though!

@fgiunchedi @herron I agree, it is a good opportunity to move it to the new infrastructure:D. Since we will be upgrading Thumbor servers to Stretch soon, we could first do the upgrade along with T212941 and then go ahead and do T212946 and T187765

@herron should I ping you when we get to the logging part? Depending on availability and needs, we can figure out how we will implement it

jijiki lowered the priority of this task from Medium to Low.Jun 21 2019, 12:05 PM

Reviving this as part of this Q's OKRs to move services off logstash non-kafka inputs, I'll followup with patches to move to the localhost-udp compatibility endpoint! (See T242609)

fgiunchedi claimed this task.

This is complete (i.e. T242609: Move thumbor to the logging pipeline), resolving. Feel free to reopen though!