Indeed, grafana alerts are to be sent to alertmanager (i.e. centrally) and from there alerts can trigger email notifications. Moving to cxserver folks, I'm assuming an alert misconfiguration (?)

FWIW as stated now the task is not actionable, I recommend setting alerts up as instructed by https://wikitech.wikimedia.org/wiki/Alertmanager#Grafana_alerts

direct email sent is not allowed, need alert manager.
All alerts sent to alert manager, need to use the puppet file to choose either email or slack or other channel,
Onboard - https://wikitech.wikimedia.org/wiki/Alertmanager#Onboard

Change #1198681 had a related patch set uploaded (by Huei Tan; author: Huei Tan):

[mediawiki/extensions/ContentTranslation@master] Remove duplicate campaign tracking

https://gerrit.wikimedia.org/r/1198681

Change #1199248 had a related patch set uploaded (by Huei Tan; author: Huei Tan):

[operations/puppet@production] alertmanager: route Language and Product Localization team alerts

https://gerrit.wikimedia.org/r/1199248

Change #1199248 merged by Andrea Denisse:

[operations/puppet@production] alertmanager: route Language and Product Localization team alerts

https://gerrit.wikimedia.org/r/1199248

Current status: example that the alert show up in the channel #lpl-feed

Next:
Once we finalize the alert rule details, i will post more details (alert rule details, config change, etc)

Change #1200053 had a related patch set uploaded (by Huei Tan; author: Huei Tan):

[operations/puppet@production] alartmanager: change the lpl-team-slack-api-alerts config

https://gerrit.wikimedia.org/r/1200053

Change #1200053 merged by Andrea Denisse:

[operations/puppet@production] alartmanager: change the lpl-team-slack-api-alerts config

https://gerrit.wikimedia.org/r/1200053

Alert rule:

Failing Events (Dashboard)
https://grafana-rw.wikimedia.org/alerting/grafana/cf2eh9yxlhibkc/view

Failing Events on Editor and Publish (Desktop)
https://grafana-rw.wikimedia.org/alerting/grafana/df2epe75if2f4e/view

Failing Events on Editor and Publish (Mobile)
https://grafana-rw.wikimedia.org/alerting/grafana/df2epm667wef4f/view

Each belongs to a separate graph in Grafana.

In T376535#11334786, @hueitan wrote:

Alert rule:

Failing Events (Dashboard)
https://grafana-rw.wikimedia.org/alerting/grafana/cf2eh9yxlhibkc/view

Failing Events on Editor and Publish (Desktop)
https://grafana-rw.wikimedia.org/alerting/grafana/df2epe75if2f4e/view

Failing Events on Editor and Publish (Mobile)
https://grafana-rw.wikimedia.org/alerting/grafana/df2epm667wef4f/view

Each belongs to a separate graph in Grafana.

Thanks Huei. 10 per hour seems to be quite low and I see that alert triggered and never resolved. How many are we getting on average? Let's get that value and set the alert value to be slightly higher than that.

Additionally, would it be possible to set up alerts for the following?

1. If there is no access to dashboard/editing/publishing event triggered in the last 30 minutes, trigger an alert.
2. If there is no publishing on desktop for the last 30 minutes, or for 3 hours on the mobile, trigger an alert.

I've set these values based on averages.

alert triggered and never resolved. How many are we getting on average? Let's get that value and set the alert value to be slightly higher than that.

I disabled the “resolve” alert because the frequent up-and-down changes were spamming our feed. I’ve switched the alerts to use averaged values now, if we see the alert persist in our feed, meaning there a real issue.

example rule

# Compares last-hour failures to the 7‑day hourly average and outputs when higher.
sum (increase(mediawiki_cx_recommendation_fail_total[1h]))
  >
sum (avg_over_time(increase(mediawiki_cx_recommendation_fail_total[1h])[7d:1h]))

Additionally, would it be possible to set up alerts for the following?

1. If there is no access to dashboard/editing/publishing event triggered in the last 30 minutes, trigger an alert.
2. If there is no publishing on desktop for the last 30 minutes, or for 3 hours on the mobile, trigger an alert.

Good suggestion, i have added the alert for
desktop access
https://grafana-rw.wikimedia.org/alerting/bf33mp3lb3caoa/edit
mobile access
https://grafana-rw.wikimedia.org/alerting/cf33n35pfgef4f/edit
publish
https://grafana-rw.wikimedia.org/alerting/cf33o16njw4xsf/edit

rule in

# zero access in the past 30 minutes
sum(
  increase(mediawiki_cx_access_articlesearch_total[30m])
+ ... // all the access items
+ increase(mediawiki_cx_access_dashboard_total[30m])
) == 0

Let's make it run for a day and see how the alert goes.

@hueitan Using a moving average as a threshold can be unreliable and lead to noisy alerts. For example, an increase from 2 to 3 errors might trigger an alert despite being acceptable. Frequent false positives will eventually reduce trust in the system.

I say we skip alerts for failures for now and setup specific threshold alerts for access events. We can revisit this later once we've had the chance to fix some of these errors.

Thanks @abi_ I have set the threshold for

threshold 25 for the alerts failures

The last hourly average - The 7 days hourly average > 25

threshold < 1 the access events

past 30 minutes no access found

This threshold doesn't spam our alert now, but provide us the alert when something goes totally wrong on the failure and access

Let's visit the failure error later then we can fix with the better threshold number