In T182497, we discovered that ORES logs are being sent to syslog, using a custom formatter. We don't know where that's coming from, we have a logging_config.yaml that explicitly sends all logging to a file.
This can be reproduced in our vagrant role. Debug and fix logging to only send to ores.log.
This might be a good opportunity to provide an /etc/ores override location for logging config, so that puppet doesn't have to write into a source-controlled directory.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | awight | T182497 Update log config for scb* boxes, to deal with ORES verbose logging | |||
| Resolved | awight | T182614 Investigate why ORES logs are being written to syslog despite explicit logging config. Fix. |
Change 393831 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/vagrant@master] [WIP] Configure ORES logging to only send to ores.log
The explanation is that Celery follows an archaic pattern of hijacking the Python root logger, using its own customizations and ignoring existing logging config. All in the interest of "having something show up on the terminal for people who don't configure logging."
There's no easy fix. I've tried everything suggested in:
Although this (theoretically) prevents Celery from damaging the root logger, it doesn't help with redirecting Celery logs per se. The least worst option I see is to monkey-patch the Celery log initialization and pass through to the default logger.
Change 398942 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@master] Clean up logging config.
Change 398942 merged by Awight:
[mediawiki/services/ores/deploy@master] Clean up logging config.
This is deployed on the beta cluster, but isn't working. I think I accidentally made us dependent on the --logging-config command line argument. The app should default to using the included logging_config.yaml if it's present.
Change 393831 merged by jenkins-bot:
[mediawiki/vagrant@master] Configure ORES logging
Mentioned in SAL (#wikimedia-operations) [2018-01-02T18:00:46Z] <awight@tin> Started deploy [ores/deploy@eb0f776]: Update ORES service to eb0f776: T182614
Mentioned in SAL (#wikimedia-operations) [2018-01-02T18:20:39Z] <awight@tin> Finished deploy [ores/deploy@eb0f776]: Update ORES service to eb0f776: T182614 (duration: 19m 55s)
Change 401781 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/vagrant@stretch-migration] Configure ORES logging