Maniphest T207834

Cleanup Wikidata Query Service logging configuration
Closed, ResolvedPublic
Actions

Description

At the moment, wdqs logging is managed by logback and is basically:

send all logs to console, to be managed by journald and sent to file
send all logs to logstash

What we should have:

immediately actionable logs should be sent to console / journald (loglevel >= WARN or ERROR)
logback should log directly to files at level => INFO for wdqs and WARN for others
logback should log directly to logstash at level => INFO for wdqs and WARN for others
wdqs-updater needs to be run manually in some cases, those logs should be identifiable (different log files / application name suffix)

Details

Project	Branch	Lines +/-	Subject
operations/puppet	production	+2 -4	wdqs: replace deprecated log appender
operations/puppet	production	+113 -28	wdqs: cleanup logback configuration
operations/puppet	production	+67 -18	wdqs: cleanup logback configuration
wikidata/query/rdf	master	+12 -4	Allow different logging configurations.

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Smalyshev	T207817 WDQS Updater ran into issue and stopped working
		Resolved		Smalyshev	T207834 Cleanup Wikidata Query Service logging configuration

Event Timeline

Gehel triaged this task as High priority.Oct 24 2018, 9:28 AM

Gehel created this task.

Restricted Application added a project: Wikidata. · View Herald TranscriptOct 24 2018, 9:28 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 463254 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: cleanup logback configuration

https://gerrit.wikimedia.org/r/463254

gerritbot added a project: Patch-For-Review.Oct 24 2018, 9:29 AM

My ideal situation for Updater logs is as follows:

Updater which is run from service logs to files, and logs important stuff (that would be mostly exceptions) to logstash. Routine stuff (INFO level) not necessary to keep in logstash.
The log files should be accessible without root access, at least to blazegraph user, ideally to all users inside wdqs-admins group.
The current log file should have constant name. The rotated logs should be marked by date.
Updater invoked from command-line logs by default to console on INFO level (it's OK to have separate option for it). logstash logging is ok but not really necessary here.
Updater invoked from command-line has an option to go to DEBUG level and another to go to TRACE level.
Neither DEBUG nor TRACE logs go to logstash.
The options above are accessible without root/sudo and not require editing config files.

For Blazegraph, we probably want everything from INFO level up into both logstash and log files. Option to run at DEBUG level would be nice but I've not used it too much in the past.

We probably should get rid of both services sending anything to journald, I am disappointed in its performance and I think we should avoid putting anything there.

My current patch is trying to put all that logic into logback.xml, but it is definitely starting to be unreadable. And coding ifs in XML just seems wrong :/

I think that instead we should have different static logback.xml files and have a flag to switch between them. Similarly to what is done with the current -v option of the updater, but via system properties. We could even have a default config in packaged in the jar (similar to the current verbose config) under the control of the application, and override it only when run as a service.

Smalyshev added a project: Wikimedia-Incident.Oct 24 2018, 8:28 PM

Smalyshev moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.

Smalyshev moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.

Smalyshev moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.Oct 24 2018, 9:35 PM

Smalyshev added a project: Discovery-Wikidata-Query-Service-Sprint.Oct 24 2018, 11:58 PM

Smalyshev moved this task from Backlog to Needs review on the Discovery-Wikidata-Query-Service-Sprint board.

Change 469611 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: cleanup logback configuration

https://gerrit.wikimedia.org/r/469611

Change 469619 had a related patch set uploaded (by Gehel; owner: Gehel):
[wikidata/query/rdf@master] Allow different logging configurations.

https://gerrit.wikimedia.org/r/469619

Change 469619 merged by jenkins-bot:
[wikidata/query/rdf@master] Allow different logging configurations.

https://gerrit.wikimedia.org/r/469619

Smalyshev moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Oct 26 2018, 8:04 PM

Change 469611 merged by Gehel:
[operations/puppet@production] wdqs: cleanup logback configuration

https://gerrit.wikimedia.org/r/469611

Change 463254 abandoned by Gehel:
wdqs: cleanup logback configuration

Reason:
superseeded by smarter approach: https://gerrit.wikimedia.org/r/c/operations/puppet/ /469611

https://gerrit.wikimedia.org/r/463254

new configuration deployed, but raising some deprecations, needs some tuning.

• Addshore moved this task from incoming to monitoring on the Wikidata board.Oct 31 2018, 5:39 PM

Change 472145 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: replace deprecated log appender

https://gerrit.wikimedia.org/r/472145

Change 472145 merged by Gehel:
[operations/puppet@production] wdqs: replace deprecated log appender

https://gerrit.wikimedia.org/r/472145

@Smalyshev we're good on this from my point of view. Could you check that running updater manually (with the -S option to output to console, or with -v for verbose) works as you expect?