In T293694#8564483, @kostajh wrote:

Is it possible to use the OpenSearch Alerting plugin for this? For context, I have arrived at this task from T328129: Implement alerting when spikes occur in Growth team dashboard in Logstash which proposes to do something similar, but for the Growth team's dashboard.

The alerting plugin is not recommended, but we have a way to feed query results into our alerting infrastructure.

Alerts on logs need a few things:

1. A query that selects and filters down to what you want to monitor
2. A defined threshold that when out of bounds triggers the alert to fire
3. Infrastructure that knows when to generate the alert and forward it to the right recipients

For (1), a dashboard gets us oriented in the right direction for developing the needed queries. For (2), we need threshold(s) and a look at the pros and cons of codifying the thresholds in VCS or the need to change it often. For (3), we use a Prometheus exporter that turns Dashboards queries into metrics, Prometheus or Grafana evaluates those metrics against the threshold(s), and AlertManager routes alerts to recipients.

In T328129#8565247, @colewhite wrote:

For (1), a dashboard gets us oriented in the right direction for developing the needed queries.

Can you use an actual dashboard, or is it more like the snapshot of a dashboard? Being able to update the dashboard's filters and have the same change automatically apply to the alerts as well would be nice. (We use the filters to ignore irrelevant known bugs. Those are usually not particularly high-volume so the filters might not be that important, but it would be a nice capability.)

For (2), we need threshold(s) and a look at the pros and cons of codifying the thresholds in VCS or the need to change it often. For (3), we use a Prometheus exporter that turns Dashboards queries into metrics, Prometheus or Grafana evaluates those metrics against the threshold(s), and AlertManager routes alerts to recipients.

If the process uses Grafana anyway, I wonder if we could just use Grafana's built-in email alerts? Thes goal of this task is more of giving us a heads-up about potential issues than emergency response, so IMO the self-serve configurability via the Grafana UI might be more useful than the robustness of having the alert rules be managed in git.

In T328129#8571676, @Tgr wrote:

Can you use an actual dashboard, or is it more like the snapshot of a dashboard? Being able to update the dashboard's filters and have the same change automatically apply to the alerts as well would be nice. (We use the filters to ignore irrelevant known bugs. Those are usually not particularly high-volume so the filters might not be that important, but it would be a nice capability.)

Currently there is no way to synchronize a set of queries from a dashboard without human intervention. The query that generates the metrics would live in Puppet and require a patch to update.

RelEng has also expressed the desire to tag certain logs as "known issues" and change views based on that property through the logstash interface: T302041: Provide way to enhance Logstash logs with information from the Train Log Triage process.

If the process uses Grafana anyway, I wonder if we could just use Grafana's built-in email alerts? Thes goal of this task is more of giving us a heads-up about potential issues than emergency response, so IMO the self-serve configurability via the Grafana UI might be more useful than the robustness of having the alert rules be managed in git.

Yes, the ability to adjust a threshold without a Gerrit patch is one advantage of this approach. SMTP is not configured for Grafana, but alerts routed through AlertManager are supported.