Page MenuHomePhabricator

Consider enabling all MW log channels by default for WMF
Open, Needs TriagePublic

Description

In T228462, we had some errors in a DeferredUpdate which is caught by DeferredUpdates::run() and logged to the DeferredUpdates channel. But the errors/exception was not showing up until @tstarling configured the logging channel in mediawiki-config:

$wmgMonologChannels['default']['DeferredUpdates'] = 'error';

If I understand it properly, that implies our log configuration must explicitly list each of the log buckets. I would have assumed we have a catch all for anything that is at error or above. Instead the channels not referenced fall back to $wmgDefaultMonologHandler which is set to blackhole logs:

wmf-config/InitialiseSettings.php
// Statically configured Monolog handler to clone for log channels that are
// not specifically configured in $wmgMonologChannels.
// See $wmgMonologConfig['handlers'] in logging.php for valid values.
'wmgDefaultMonologHandler' => [
    'default' => 'blackhole',
    'testwiki' => 'wgDebugLogFile',
    'test2wiki' => 'wgDebugLogFile',
],
wmf-config/logging.php
$wmgMonologConfig = [
    'loggers' => [
        // Template for all undefined log channels
        '@default' => [
            'handlers' => (array)$wmgDefaultMonologHandler, 
            'processors' => array_keys( $wmgMonologProcessors ),
            'calls' => $wmgMonologLoggerCalls,
        ],
    ],

So unless a log channel is explicitly referenced, we are missing errors :-\ I guess that instead of using a black hole, we should at least log at error level, if not at warning level. Maybe with an extra context field to differentiate those fallback logs.

Event Timeline

hashar created this task.Jul 24 2019, 8:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 24 2019, 8:26 AM
Tgr added a comment.Jul 24 2019, 9:02 AM

+1, I always found this very annoying. I think it was originally done because Logstash was formally unsupported and not well resourced, and we had to be cautious not to overload it, but that shouldn't be the case anymore.

I always thought the convention of dropping unknown channels was just carried over from the pre-logstash system. Before Monolog, we only had wfDebugLog(), and so the only way to distinguish between debug channels and error channels was by channel name. Dropping unknown channels avoided accidentally DoSing the logging system when a new debug channel was added.

Krinkle renamed this task from Unreferenced log channels are not logged at all causing "logged" errors to be missed to Consider enabling all MW log channels by default for WMF.Tue, Aug 27, 2:14 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Wed, Aug 28, 11:05 PM