Page MenuHomePhabricator

Cleanup Wikidata Query Service logging configuration
Closed, ResolvedPublic

Description

At the moment, wdqs logging is managed by logback and is basically:

  • send all logs to console, to be managed by journald and sent to file
  • send all logs to logstash

What we should have:

  • immediately actionable logs should be sent to console / journald (loglevel >= WARN or ERROR)
  • logback should log directly to files at level => INFO for wdqs and WARN for others
  • logback should log directly to logstash at level => INFO for wdqs and WARN for others
  • wdqs-updater needs to be run manually in some cases, those logs should be identifiable (different log files / application name suffix)

Event Timeline

Gehel created this task.Oct 24 2018, 9:28 AM
Gehel triaged this task as High priority.
Restricted Application added a project: Wikidata. · View Herald TranscriptOct 24 2018, 9:28 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 463254 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: cleanup logback configuration

https://gerrit.wikimedia.org/r/463254

Smalyshev added a subscriber: Smalyshev.EditedOct 24 2018, 4:12 PM

My ideal situation for Updater logs is as follows:

  • Updater which is run from service logs to files, and logs important stuff (that would be mostly exceptions) to logstash. Routine stuff (INFO level) not necessary to keep in logstash.
  • The log files should be accessible without root access, at least to blazegraph user, ideally to all users inside wdqs-admins group.
  • The current log file should have constant name. The rotated logs should be marked by date.
  • Updater invoked from command-line logs by default to console on INFO level (it's OK to have separate option for it). logstash logging is ok but not really necessary here.
  • Updater invoked from command-line has an option to go to DEBUG level and another to go to TRACE level.
  • Neither DEBUG nor TRACE logs go to logstash.
  • The options above are accessible without root/sudo and not require editing config files.

For Blazegraph, we probably want everything from INFO level up into both logstash and log files. Option to run at DEBUG level would be nice but I've not used it too much in the past.

We probably should get rid of both services sending anything to journald, I am disappointed in its performance and I think we should avoid putting anything there.

Gehel added a comment.Oct 24 2018, 4:26 PM

My current patch is trying to put all that logic into logback.xml, but it is definitely starting to be unreadable. And coding ifs in XML just seems wrong :/

I think that instead we should have different static logback.xml files and have a flag to switch between them. Similarly to what is done with the current -v option of the updater, but via system properties. We could even have a default config in packaged in the jar (similar to the current verbose config) under the control of the application, and override it only when run as a service.

Change 469611 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: cleanup logback configuration

https://gerrit.wikimedia.org/r/469611

Change 469619 had a related patch set uploaded (by Gehel; owner: Gehel):
[wikidata/query/rdf@master] Allow different logging configurations.

https://gerrit.wikimedia.org/r/469619

Change 469619 merged by jenkins-bot:
[wikidata/query/rdf@master] Allow different logging configurations.

https://gerrit.wikimedia.org/r/469619

Change 469611 merged by Gehel:
[operations/puppet@production] wdqs: cleanup logback configuration

https://gerrit.wikimedia.org/r/469611

Change 463254 abandoned by Gehel:
wdqs: cleanup logback configuration

Reason:
superseeded by smarter approach: https://gerrit.wikimedia.org/r/c/operations/puppet/ /469611

https://gerrit.wikimedia.org/r/463254

Gehel added a comment.Oct 30 2018, 5:35 PM

new configuration deployed, but raising some deprecations, needs some tuning.

Addshore moved this task from incoming to monitoring on the Wikidata board.Oct 31 2018, 5:39 PM

Change 472145 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: replace deprecated log appender

https://gerrit.wikimedia.org/r/472145

Change 472145 merged by Gehel:
[operations/puppet@production] wdqs: replace deprecated log appender

https://gerrit.wikimedia.org/r/472145

@Smalyshev we're good on this from my point of view. Could you check that running updater manually (with the -S option to output to console, or with -v for verbose) works as you expect?

Smalyshev closed this task as Resolved.Nov 7 2018, 4:38 PM
Smalyshev claimed this task.

I still need a config for enabling TRACE level, but the rest seems to work fine.