Seems to affect everything that's type:mediawiki, channel:exception. Breaks the "most frequent" feature on the dashboard.
|operations/puppet : production||logstash: Remove filter for unused 'exception-json' channel|
|operations/mediawiki-config : master||Do not send 'exception-json' channel to logstash|
|operations/mediawiki-config : master||Send 'exception' channel to logstash|
|operations/puppet : production||logstash: Add processing rules for MediaWiki's exception channel|
|Open||None||T157850 Interacting with Wikimedia logs should be a pleasant experience|
|Resolved||None||T136849 normalized_message is a JSON dump of the whole event for exceptions in beta logstash|
- Mentioned In
- T45086: Capture PHP warnings with stacktraces in MediaWiki and save to logstash
T154112: Editing session issue (TitleBlacklist related?) on *.wikipedia.org for Kvardek du
- Mentioned Here
- P4968 (An Untitled Masterwork)
P4969 (An Untitled Masterwork)
T151290: When logging an exception in Logstash as a PSR-3 context parameter, the trace does not include class/method names
Having that data in a structured form would actually be quite nice as it would allow e.g. filtering on the exception class. But normalized_message is truncated to some character limit so it's not really useful.
This is caused by the custom normalized_message processor being unable to properly handle the exception-json channel which consists of events with no context array a stringified JSON blob as message.
@bd808 what's the reason we use that format instead of properly putting the data in the context array like the normal exception channel does? Is that for the benefit of the pre-Monolog handler?
It's stuff that @Krinkle put into core before we had structured logging. I remember using that channel instead of the normal exception channel when ELK was first setup and being fed from the udp2log tap. I thought I switched back to the exception channel in the Logstash config when we got Monolog fully wired into the WMF log stack, but maybe I only meant to and never actually did that?
- normalized_message is fixed, yay!
- backtrace is replaced by exception.trace. The format is not great, T151290 should add human-readable traces, but maybe we want the old structured format as well? This will break the trending backtrace file visualization.
- message (and normalized_message) is more cluttered - before it was just the message, now it's the text rendered by the exception (including hash, URL, file/line, exception class). (The "pure" message is available as exception.message.) I don't mind file/line/class but having the URL in there will breaks aggregation.
- exception class wasn't available before, now it is in exception.class. That's nice and should be added to the dashboards.
- some fields moved around (file & line -> exception.file, code -> exception.code). Hopefully does not break anything.
- private flag is gone (should not be missed)
message could be fixed either in LogstashFormatter or in the logstash config. The first is a mildly horrible hack, the second needs to be written in custom symtax and is a lot more effort to test.