The sawtooth pattern reminded me of T395899. It shouldn't be related, but maybe there was a similar throttle somewhere affecting these logs.

The log volume seems to have gone mostly back to normal on Monday afternoon. Here's the same search, but with a bigger chart: https://logstash.wikimedia.org/goto/35a757159907937d81e30015adb08663

In T404546#11191089, @matmarex wrote:

The sawtooth pattern reminded me of T395899. It shouldn't be related, but maybe there was a similar throttle somewhere affecting these logs.

The post-kafka throttle/ratelimit described in T395899 is not configured to affect these logs. The throttler, prior to dropping logs will tag the events with throttle_warning and still be found in logstash. There are no throttle_warning events in that query.

We observed no logging system outages during that time. Just prior to recovery, however, there was a noticeable dip in proxy/httpd logs flowing towards logstash.