Logstash started showing full serialized log entry as a message
Closed, ResolvedPublic

Description

As of aroung 16:20 UTC on Jun 13 for some reason logstash started showing complete log entry serialized as JSON in the message field instead of the actual message taken from the log entry.

Example for the same type of the log message:

16:22 https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.06.13/cpjobqueue?id=AWP59E_WoOODFPKv9AEI&_g=h@44136fa
16:21 https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.06.13/cpjobqueue?id=AWP589O3oOODFPKv8_a7&_g=h@b9a3c59

This happened around the same time for all node services and non of them was deployed around that time, so I'm pretty sure the problem is not in the services themselves.

Pchelolo created this task.Jun 14 2018, 9:04 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 14 2018, 9:04 AM
Joe added a subscriber: Joe.Jun 18 2018, 1:07 PM

At a quick glance, neither Mediawiki-generated logs nor syslog generated ones show this issue. I can't find anything relevant in the SAL, but I'll try to dig deeper.

Joe claimed this task.Jun 18 2018, 1:22 PM

Change 440861 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] role::logstash: fix gelf filtering

https://gerrit.wikimedia.org/r/440861

Change 440861 merged by Giuseppe Lavagetto:
[operations/puppet@production] role::logstash: fix gelf filtering

https://gerrit.wikimedia.org/r/440861

Joe added a comment.Jun 18 2018, 2:20 PM

So after some reasoning:

  • elasticsearch needs to use full_message as short message is truncated
  • the node applications need to use short_message

Things should be ok as of now, I'm resolving the ticket but feel free to reopen it if you still notice things going south.

Joe closed this task as Resolved.Jun 18 2018, 2:20 PM
Joe triaged this task as High priority.
Vvjjkkii renamed this task from Logstash started showing full serialized log entry as a message to 00aaaaaaaa.Jul 1 2018, 1:04 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Joe as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
mobrovac renamed this task from 00aaaaaaaa to Logstash started showing full serialized log entry as a message.Jul 1 2018, 11:34 AM
mobrovac closed this task as Resolved.
mobrovac assigned this task to Joe.
mobrovac updated the task description. (Show Details)
mobrovac removed a subscriber: Joe.