Page MenuHomePhabricator

WDQS logging should be rate limited
Closed, ResolvedPublic

Description

We've had a few issues lately where Wikidata Query Service was overloading logstash. While we already have some mechanisms in place to limit the amount of logging generated by wdqs, they are not robust enough.

Having a rate limite on the logstash appender should ensure that we don't overload logstash. It will have to drop logging events, so we will loose informatin, but that is better than overloading logstash, still loosing information and worse: loosing information from other unrelated applications.

Rate limiting should be applied to most (if not all) appenders. Overloading journald is also an issue, as seen during T207817.

Note that we still need to tune logging to ensure that this limit is not reached. This rate limit is a last barrier if all other means have failed.

Event Timeline

Gehel created this task.Oct 22 2018, 1:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2018, 1:44 PM
Gehel triaged this task as High priority.Oct 22 2018, 1:46 PM
Gehel updated the task description. (Show Details)

Change 468977 had a related patch set uploaded (by Gehel; owner: Gehel):
[wikidata/query/rdf@master] logback rate limit filter

https://gerrit.wikimedia.org/r/468977

Change 468977 merged by jenkins-bot:
[wikidata/query/rdf@master] logback rate limit filter

https://gerrit.wikimedia.org/r/468977

Change 468979 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: rate limit log sent to logstash

https://gerrit.wikimedia.org/r/468979

Gehel renamed this task from WDQS logging to logstash should be rate limited to WDQS logging should be rate limited.Oct 24 2018, 7:49 AM
Gehel updated the task description. (Show Details)

Change 468979 merged by Gehel:
[operations/puppet@production] wdqs: rate limit log sent to logstash

https://gerrit.wikimedia.org/r/468979

Smalyshev closed this task as Resolved.Nov 7 2018, 4:38 PM
Smalyshev claimed this task.