Page MenuHomePhabricator

Email/Slack notification when job fails
Open, Needs TriagePublic

Description

If the red box on https://pixel.wmcloud.org/reports/desktop/index.html shows > 0 errors, I would like it to send an email to reading-web-team@lists.wikimedia.org to tell us.

This would work similar to our grafana alerting to allow us to respond.

./pixel.js test -email-on-error reading-web-team@lists.wikimedia.org

Event Timeline

Jdlrobson-WMF renamed this task from Email when job fails to Email/Slack notification when job fails.Feb 2 2026, 9:35 PM
Jdlrobson-WMF subscribed.

If we could route this to the internal #performance-alerts channel that would be helpful.

@Peter @Mhurd is this something on your roadmap for the forseeable future? It's increasingly becoming a need for readers teams given a few recent regressions.

We have the data in https://grafana.wikimedia.org/d/lC3anj1Iz/pixel?orgId=1&from=now-2d&to=now&timezone=utc so easiest would be to send the alerts Grafana, there there's no work on our side (but looking at the graphs it looks like it will be chatty).

In https://phabricator.wikimedia.org/T406689 it seems like there was a routing added for sending to Slack, maybe that can be used?

I was going to file a feature request for having pixel failures surfaced on phabricator as new tasks (similar to other tooling), as I find visibility a big problem with the current version of pixel. Surely I could file a separate task, but instead, could this task be repurposed to implement phab integration instead, in the spirit of favouring a more public and transparent solution? Slack/email integration could then be implemented by those who want it via a bot that listens to new tasks (see also the existing slack_connector_* bots).

We have the data in https://grafana.wikimedia.org/d/lC3anj1Iz/pixel?orgId=1&from=now-2d&to=now&timezone=utc so easiest would be to send the alerts Grafana, there there's no work on our side (but looking at the graphs it looks like it will be chatty).

Neato. I'll look into that shortly. Assuming I can send notifications on a per-job basis than I likely have what I need to set this up (I'll also get rid of the noise as part of this so will likely post a pixel patch).

could this task be repurposed

Would you mind opening a new ticket as I'll likely have some follow up questions as I implement.

could this task be repurposed

Would you mind opening a new ticket as I'll likely have some follow up questions as I implement.

Done: T416524: Automatically report pixel failures in phabricator