Page MenuHomePhabricator

Logstash started showing full serialized log entry as a message
Closed, ResolvedPublic

Description

As of aroung 16:20 UTC on Jun 13 for some reason logstash started showing complete log entry serialized as JSON in the message field instead of the actual message taken from the log entry.

Example for the same type of the log message:

16:22 https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.06.13/cpjobqueue?id=AWP59E_WoOODFPKv9AEI&_g=h@44136fa
16:21 https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.06.13/cpjobqueue?id=AWP589O3oOODFPKv8_a7&_g=h@b9a3c59

This happened around the same time for all node services and non of them was deployed around that time, so I'm pretty sure the problem is not in the services themselves.

Event Timeline

At a quick glance, neither Mediawiki-generated logs nor syslog generated ones show this issue. I can't find anything relevant in the SAL, but I'll try to dig deeper.

Change 440861 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] role::logstash: fix gelf filtering

https://gerrit.wikimedia.org/r/440861

Change 440861 merged by Giuseppe Lavagetto:
[operations/puppet@production] role::logstash: fix gelf filtering

https://gerrit.wikimedia.org/r/440861

So after some reasoning:

  • elasticsearch needs to use full_message as short message is truncated
  • the node applications need to use short_message

Things should be ok as of now, I'm resolving the ticket but feel free to reopen it if you still notice things going south.

Joe triaged this task as High priority.
Vvjjkkii renamed this task from Logstash started showing full serialized log entry as a message to 00aaaaaaaa.Jul 1 2018, 1:04 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Joe as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
mobrovac renamed this task from 00aaaaaaaa to Logstash started showing full serialized log entry as a message.Jul 1 2018, 11:34 AM
mobrovac closed this task as Resolved.
mobrovac assigned this task to Joe.
mobrovac updated the task description. (Show Details)
mobrovac removed a subscriber: Joe.