Page MenuHomePhabricator

Introduce logging pipeline input ratelimit
Open, Needs TriagePublic

Description

As experienced in T215611: MediaWiki errors overloading logstash logging pipeline users can overload the pipeline itself, causing trouble/slowdowns in unrelated applications too. We should thus introduce rate limits for admission control of logs, to avoid application-driven spam. In this context 'spam' refers to an unusually high amount of logs either related to an external cause (e.g. application dependency down) or an internal one (bug in the application causing too much logging)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 12 2019, 2:15 PM
Gehel added a subscriber: Gehel.Feb 15 2019, 1:33 PM

Slightly related to this topic, we've had similar issues on WDQS where the amount of logging went out of control. To address the issue at WDQS level, we implemented to filters (PerLoggerThrottler and RateLimitFilter), which can be used to limit the amount of logs sent by the application. Those can be reused on a Java / logback stack if needed. And of course, while an application should be well behaved, the logging pipeline should not assume that it is the case.