While investigating a thanos-query alert today I noticed warnings logged by thanos-query with very long lines in the journal and local syslog files on thanos-fe hosts.
These seem like a good candidate for aggregation in logstash, and based on the local syslog file sizes the volume doesn't seem like it would be problematic.
Let's discuss bringing these into logstash.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| logstash: output thanos-query syslogs to kafka and local file | operations/puppet | production | +2 -1 |
Change 828960 had a related patch set uploaded (by Herron; author: Herron):
[operations/puppet@production] logstash: output thanos-query syslogs to kafka and local file
Change 828960 merged by Herron:
[operations/puppet@production] logstash: output thanos-query syslogs to kafka and local file
+1
The logs appear very similar in format to prometheus-blackbox-exporter: that is logfmt. It'd be great to get them to follow the same processing path: rsyslog apply ecs_170 in lookup_table_output and use an early-stage filter like blackbox exporter (possibility for filter reuse here?)
This would get it into the ECS pipeline. Also, some tests to help lock it in.