For example:
https://logstash.wikimedia.org/goto/35937df5bb390e3ff2cb06e42a83cac7 --- almost 2M messages in 3.3h
What is odd is that the log message is just %{message} as I understand it. If so, that would indicate a broken log line.
This originally happened at the beginning of April, on staging. It may well be that the bad code is already gone.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T391943 Q4 24-25 Goal: Operational Excellence - LiftWing Platform Updates & Improvements | |||
| Open | None | T398948 Q1 FY2025-26 Goal: Operational Excellence - LiftWing Platform Updates & Improvements | |||
| Open | isarantopoulos | T393475 ML Services causing log spam |
This issue seemed to be transient and was likely caused while running load testing on ml-staging without disabling logging.
Resolving this for now.
cc: @Ladsgroup
It seems it started log flooding again. Here is just the staging causing 250K logs in 15 minutes: https://logstash.wikimedia.org/goto/f2c444ed44f453d25636ad25739a0897
If I'm reading logs correctly, 50% of all logs are because of ML hosts (100K logs per minute). This is just a couple of hosts: https://logstash.wikimedia.org/goto/43a05b927e05c87617a1c498ce1e9f78
Thanks for reporting! These messages seem to come from many servers (prod & staging) and from multiple model servers. So it was not related to load testing on ml-staging. Also not all of them seem to be just %{message} (although a lot of them are)
@klausman can we get any more info on this ?