Page MenuHomePhabricator

Investigate why ORES logs are being written to syslog despite explicit logging config. Fix.
Closed, ResolvedPublic

Description

In T182497, we discovered that ORES logs are being sent to syslog, using a custom formatter. We don't know where that's coming from, we have a logging_config.yaml that explicitly sends all logging to a file.

This can be reproduced in our vagrant role. Debug and fix logging to only send to ores.log.

This might be a good opportunity to provide an /etc/ores override location for logging config, so that puppet doesn't have to write into a source-controlled directory.

Details

Related Gerrit Patches:
mediawiki/vagrant : stretch-migrationConfigure ORES logging
mediawiki/vagrant : masterConfigure ORES logging
mediawiki/services/ores/deploy : masterClean up logging config.

Event Timeline

awight renamed this task from Investigate why ORES logs are being written to syslog despite explicit logging config to Investigate why ORES logs are being written to syslog despite explicit logging config. Fix..Dec 11 2017, 6:24 PM
awight triaged this task as Medium priority.
awight created this task.
awight updated the task description. (Show Details)

Change 393831 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/vagrant@master] [WIP] Configure ORES logging to only send to ores.log

https://gerrit.wikimedia.org/r/393831

Halfak added a subscriber: Halfak.Dec 11 2017, 10:05 PM

The explanation is that Celery follows an archaic pattern of hijacking the Python root logger, using its own customizations and ignoring existing logging config. All in the interest of "having something show up on the terminal for people who don't configure logging."

There's no easy fix. I've tried everything suggested in:

Although this (theoretically) prevents Celery from damaging the root logger, it doesn't help with redirecting Celery logs per se. The least worst option I see is to monkey-patch the Celery log initialization and pass through to the default logger.

Change 398942 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@master] Clean up logging config.

https://gerrit.wikimedia.org/r/398942

Change 398942 merged by Awight:
[mediawiki/services/ores/deploy@master] Clean up logging config.

https://gerrit.wikimedia.org/r/398942

This is deployed on the beta cluster, but isn't working. I think I accidentally made us dependent on the --logging-config command line argument. The app should default to using the included logging_config.yaml if it's present.

Works perfectly now. Waiting until January to deploy to production.

Change 393831 merged by jenkins-bot:
[mediawiki/vagrant@master] Configure ORES logging

https://gerrit.wikimedia.org/r/393831

Mentioned in SAL (#wikimedia-operations) [2018-01-02T18:00:46Z] <awight@tin> Started deploy [ores/deploy@eb0f776]: Update ORES service to eb0f776: T182614

Mentioned in SAL (#wikimedia-operations) [2018-01-02T18:20:39Z] <awight@tin> Finished deploy [ores/deploy@eb0f776]: Update ORES service to eb0f776: T182614 (duration: 19m 55s)

awight added a comment.Jan 2 2018, 6:25 PM
This comment was removed by awight.

Change 401781 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/vagrant@stretch-migration] Configure ORES logging

https://gerrit.wikimedia.org/r/401781

Change 401781 abandoned by Awight:
Configure ORES logging

https://gerrit.wikimedia.org/r/401781

Halfak closed this task as Resolved.Jan 30 2018, 8:31 PM