Page MenuHomePhabricator

Log suppressed errors with level=DEBUG
Open, MediumPublic


We currently discard errors that are suppressed via error_reporting(). That's problematic when debugging, as sometimes those are legitimate errors that are causing the operation to fail. They should be logged with level=DEBUG instead.

Current status
  • Logging for suppressed error has worked for years, but goes to the error-json channel instead of error. We want to deprecate that channel so that its messages can be consumed during local development (and by third parties) in a way that is compatible and similar to all other ways we log messages from MediaWiki - not in a custom json format. This will also allow us to remove custom configuration from out Logstash filter in Puppet, which currently exists only for this channel.
  • Before we can move message from error-json to error (as level=DEBUG) we first need to have a way to easily configure wgDebugLogGroups (or sometihng like that) on a plain MW install in a way that only puts messages in a given file for a given channel above a given threshold. Thus enabling developers and CI to keep this file limited to important errors. In particular because CI enforces error-free by asserting that error.log stays empty.
  • These errors currently log at their original (unsuppressed) severity level, which means they show up in general MediaWiki-Logstash queries for errors, which can be confusing.

Event Timeline

Change 338901 had a related patch set uploaded (by Gergő Tisza; owner: Krinkle):
[mediawiki/core@master] exception: Add suppressed errors to "error" channel as level=DEBUG

The patch breaks CI which checks that nothing gets logged on the error channel ( Not sure if there's an easy way to fix that in CI config, or we need to reconsider the decision to use severity levels instead of a dedicated channel.

CommunityTechBot raised the priority of this task from High to Needs Triage.Jul 3 2018, 1:53 AM

@Tgr The task as written appears to be working as expected. We currently, and have for years, collect the "suppressed errors" in production as well. They just go to a different channel (error-json, also stored in Logstash).

As stated on Gerrit, I do agree with the direction of merging the channels and using severity levels instead. But that's currently blocked on easily making that work out of the box for third-parties as well as our CI requirements.

Current WMF config discards error-json entirely.

I see your point about this change being problematic with the default config (it does seem to work fine for our CI now). Does that mean this is blocked on T155552: Make Monolog the default debug processing layer and deprecate wfDebug* and LegacyLogger? Or should we have an error-supressed channel instead of just severities, which can be handled easily with the default setup?

@Tgr That seems fine by me as a first step (separate channel).

@hashar Yeah, although it's not a "bug". This was designed on purpose, and T228838 is about changing that. For this task, if we want to collect the "error-json" channel in production, we can just enable that in wmf-config. We do all the time when introducing new channels, no big deal.

This task is about changing AtEase messages to not go into "error-json" anymore, but instead go into the main "error" channel with "level=DEBUG". Making that change is currently blocked because of two things:

  • We want to strictly prevent commits from merging that cause errors. Which is why Jenkins fails commits with a non-empty "error" channel file. If we start sending errors that were intentionally handled ("caught") into that file as well, the build would fail (and we see this already at
  • In production, we use Monolog to configure for each channel what the minimum severity level is to decide whether to send a message to a file/Logstash. We probably need to make this part of MediaWiki core by default in an easy to use way, so that we can update CI config from $wgDebugLogGroups['error'] = mw-error.log to something like $wgDebugLogGroups['error'] !? severity=INFO !? … mw-error.log. This is not limited to CI, it would help with general development and for third-party sysadmins as well. Right now the way we configure log channels in production requires a ton of complex configuration that I don't think is reasonable to expect someone to whip up in LocalSettings.php themselves.
Krinkle triaged this task as Medium priority.Aug 27 2019, 2:22 PM

@Tgr Right now, "error-json" is not enabled in wmf-config which means, except for X-Wikimedia-Debug/log requests, these are not sent to Logstash. Do you think their volume is low enough to be acceptable unconditionally? Given they usually caught/suppressed on purpose, it would mean that the 99% of requests (guesstimate) that currently don't report anything to Logstash, would have at least one or two messages at minimum under normal conditions. Years ago when we sometimes introduced something like a PHP error caused by a typo in wmf-config, it led to Logstash problems, but, maybe things are better now?

Change 553528 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] exception: Use level=DEBUG for error-json (surpressed errors)

Krinkle added a subscriber: Anomie.

Change 553528 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] exception: Use level=DEBUG for error-json (surpressed errors)

@Anomie Could you review this one? This would make the error-json messages about intentionally surpressed errors arrive in Logstash with level:DEBUG instead of e.g. WARNING. This in turn helps reduce noise in team and product level dashboards in Kibana by being able to simply query for type:mediawiki exception.trace:something -level:DEBUG|INFO and not need any awareness of e.g. "erorr-json" being a channel to ignore.

Change 553528 merged by jenkins-bot:
[mediawiki/core@master] exception: Use level=DEBUG for error-json (surpressed errors)

@Anomie Could you review this one?

Looks like Gergő beat me to it.