Introduction
On the Abstract Wikipedia team, we have our chores, which is our own flavor of "engineering observability rounds". Basically, engineers monitor various sources (like Logstash, Grafana, etc.) to search for anomalies and collect data on the state of our systems. This is usually done once per day.
While the alerts are useful for many of those things, I am trying to automate some of the discovery part of that work (things we do not have alerts for because they are new, or situations that haven't happened before, so we do not have regular alerts to detect them).
Problem statement
My initial take on this problem is to write a small script to check for some of the errors and warnings in our logs, using the Logstash API and, from there, see what we can do (and what we can't), such as aggregating errors and warnings and, if possible, reporting them somewhere.
Given the restrictions on the log machines, probably that script should be somewhere with access to those machines (I don't know where). What are your thoughts on this? Can you help me with a proper setup for this?