As experienced in T215611: MediaWiki errors overloading logstash logging pipeline users can overload the pipeline itself, causing trouble/slowdowns in unrelated applications too. We should thus introduce rate limits for admission control of logs, to avoid application-driven spam. In this context 'spam' refers to an unusually high amount of logs either related to an external cause (e.g. application dependency down) or an internal one (bug in the application causing too much logging)
Description
Description
Related Objects
Related Objects
- Mentioned In
- T215904: Better understanding of Logstash performance
- Mentioned Here
- T215611: MediaWiki errors overloading logstash
Event Timeline
Comment Actions
Slightly related to this topic, we've had similar issues on WDQS where the amount of logging went out of control. To address the issue at WDQS level, we implemented to filters (PerLoggerThrottler and RateLimitFilter), which can be used to limit the amount of logs sent by the application. Those can be reused on a Java / logback stack if needed. And of course, while an application should be well behaved, the logging pipeline should not assume that it is the case.