Page MenuHomePhabricator

Log suppressed errors with level=DEBUG
Open, MediumPublic

Description

We currently discard errors that are suppressed via error_reporting(). That's problematic when debugging, as sometimes those are legitimate errors that are causing the operation to fail. They should be logged with level=DEBUG instead.

Event Timeline

Tgr created this task.May 1 2018, 9:52 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 1 2018, 9:52 AM

Change 338901 had a related patch set uploaded (by Gergő Tisza; owner: Krinkle):
[mediawiki/core@master] exception: Add suppressed errors to "error" channel as level=DEBUG

https://gerrit.wikimedia.org/r/338901

The patch breaks CI which checks that nothing gets logged on the error channel (https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit-jessie/62535/console). Not sure if there's an easy way to fix that in CI config, or we need to reconsider the decision to use severity levels instead of a dedicated channel.

Vvjjkkii renamed this task from Log suppressed errors with level=DEBUG to 3wdaaaaaaa.Jul 1 2018, 1:12 AM
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
CommunityTechBot raised the priority of this task from High to Needs Triage.Jul 3 2018, 1:53 AM

@Tgr The task as written appears to be working as expected. We currently, and have for years, collect the "suppressed errors" in production as well. They just go to a different channel (error-json, also stored in Logstash).

As stated on Gerrit, I do agree with the direction of merging the channels and using severity levels instead. But that's currently blocked on easily making that work out of the box for third-parties as well as our CI requirements.

Tgr added a comment.Dec 9 2018, 6:24 PM

Current WMF config discards error-json entirely.

I see your point about this change being problematic with the default config (it does seem to work fine for our CI now). Does that mean this is blocked on T155552: Make Monolog the default debug processing layer and deprecate wfDebug* and LegacyLogger? Or should we have an error-supressed channel instead of just severities, which can be handled easily with the default setup?

@Tgr That seems fine by me as a first step (separate channel).

hashar added a subscriber: hashar.Aug 27 2019, 12:29 PM

Might be the same issue as T228838

@hashar Yeah, although it's not a "bug". This was designed on purpose, and T228838 is about changing that. For this task, if we want to collect the "error-json" channel in production, we can just enable that in wmf-config. We do all the time when introducing new channels, no big deal.

This task is about changing AtEase messages to not go into "error-json" anymore, but instead go into the main "error" channel with "level=DEBUG". Making that change is currently blocked because of two things:

  • We want to strictly prevent commits from merging that cause errors. Which is why Jenkins fails commits with a non-empty "error" channel file. If we start sending errors that were intentionally handled ("caught") into that file as well, the build would fail (and we see this already at https://gerrit.wikimedia.org/r/338901).
  • In production, we use Monolog to configure for each channel what the minimum severity level is to decide whether to send a message to a file/Logstash. We probably need to make this part of MediaWiki core by default in an easy to use way, so that we can update CI config from $wgDebugLogGroups['error'] = mw-error.log to something like $wgDebugLogGroups['error'] !? severity=INFO !? … mw-error.log. This is not limited to CI, it would help with general development and for third-party sysadmins as well. Right now the way we configure log channels in production requires a ton of complex configuration that I don't think is reasonable to expect someone to whip up in LocalSettings.php themselves.
Krinkle triaged this task as Medium priority.Aug 27 2019, 2:22 PM
Krinkle added a comment.EditedAug 27 2019, 2:25 PM

@Tgr Right now, "error-json" is not enabled in wmf-config which means, except for X-Wikimedia-Debug/log requests, these are not sent to Logstash. Do you think their volume is low enough to be acceptable unconditionally? Given they usually caught/suppressed on purpose, it would mean that the 99% of requests (guesstimate) that currently don't report anything to Logstash, would have at least one or two messages at minimum under normal conditions. Years ago when we sometimes introduced something like a PHP error caused by a typo in wmf-config, it led to Logstash problems, but, maybe things are better now?

Change 553528 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] exception: Use level=DEBUG for error-json (surpressed errors)

https://gerrit.wikimedia.org/r/553528