We currently discard errors that are suppressed via error_reporting(). That's problematic when debugging, as sometimes those are legitimate errors that are causing the operation to fail. They should be logged with level=DEBUG instead.
|mediawiki/core : master||exception: Use level=DEBUG for error-json (surpressed errors)|
|mediawiki/core : master||exception: Add suppressed errors to "error" channel as level=DEBUG|
|Open||None||T157850 Interacting with Wikimedia logs should be a pleasant experience|
|Open||None||T193472 Log suppressed errors with level=DEBUG|
The patch breaks CI which checks that nothing gets logged on the error channel (https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit-jessie/62535/console). Not sure if there's an easy way to fix that in CI config, or we need to reconsider the decision to use severity levels instead of a dedicated channel.
@Tgr The task as written appears to be working as expected. We currently, and have for years, collect the "suppressed errors" in production as well. They just go to a different channel (error-json, also stored in Logstash).
As stated on Gerrit, I do agree with the direction of merging the channels and using severity levels instead. But that's currently blocked on easily making that work out of the box for third-parties as well as our CI requirements.
Current WMF config discards error-json entirely.
I see your point about this change being problematic with the default config (it does seem to work fine for our CI now). Does that mean this is blocked on T155552: Make Monolog the default debug processing layer and deprecate wfDebug* and LegacyLogger? Or should we have an error-supressed channel instead of just severities, which can be handled easily with the default setup?
@hashar Yeah, although it's not a "bug". This was designed on purpose, and T228838 is about changing that. For this task, if we want to collect the "error-json" channel in production, we can just enable that in wmf-config. We do all the time when introducing new channels, no big deal.
This task is about changing AtEase messages to not go into "error-json" anymore, but instead go into the main "error" channel with "level=DEBUG". Making that change is currently blocked because of two things:
- We want to strictly prevent commits from merging that cause errors. Which is why Jenkins fails commits with a non-empty "error" channel file. If we start sending errors that were intentionally handled ("caught") into that file as well, the build would fail (and we see this already at https://gerrit.wikimedia.org/r/338901).
- In production, we use Monolog to configure for each channel what the minimum severity level is to decide whether to send a message to a file/Logstash. We probably need to make this part of MediaWiki core by default in an easy to use way, so that we can update CI config from $wgDebugLogGroups['error'] = mw-error.log to something like $wgDebugLogGroups['error'] !? severity=INFO !? … mw-error.log. This is not limited to CI, it would help with general development and for third-party sysadmins as well. Right now the way we configure log channels in production requires a ton of complex configuration that I don't think is reasonable to expect someone to whip up in LocalSettings.php themselves.
@Tgr Right now, "error-json" is not enabled in wmf-config which means, except for X-Wikimedia-Debug/log requests, these are not sent to Logstash. Do you think their volume is low enough to be acceptable unconditionally? Given they usually caught/suppressed on purpose, it would mean that the 99% of requests (guesstimate) that currently don't report anything to Logstash, would have at least one or two messages at minimum under normal conditions. Years ago when we sometimes introduced something like a PHP error caused by a typo in wmf-config, it led to Logstash problems, but, maybe things are better now?