If the red box on https://pixel.wmcloud.org/reports/desktop/index.html shows > 0 errors, I would like it to send an email to reading-web-team@lists.wikimedia.org to tell us.
This would work similar to our grafana alerting to allow us to respond.
./pixel.js test -email-on-error reading-web-team@lists.wikimedia.org
If we could route this to the internal #performance-alerts channel that would be helpful.
We have the data in https://grafana.wikimedia.org/d/lC3anj1Iz/pixel?orgId=1&from=now-2d&to=now&timezone=utc so easiest would be to send the alerts Grafana, there there's no work on our side (but looking at the graphs it looks like it will be chatty).
In https://phabricator.wikimedia.org/T406689 it seems like there was a routing added for sending to Slack, maybe that can be used?
I was going to file a feature request for having pixel failures surfaced on phabricator as new tasks (similar to other tooling), as I find visibility a big problem with the current version of pixel. Surely I could file a separate task, but instead, could this task be repurposed to implement phab integration instead, in the spirit of favouring a more public and transparent solution? Slack/email integration could then be implemented by those who want it via a bot that listens to new tasks (see also the existing slack_connector_* bots).
We have the data in https://grafana.wikimedia.org/d/lC3anj1Iz/pixel?orgId=1&from=now-2d&to=now&timezone=utc so easiest would be to send the alerts Grafana, there there's no work on our side (but looking at the graphs it looks like it will be chatty).
Neato. I'll look into that shortly. Assuming I can send notifications on a per-job basis than I likely have what I need to set this up (I'll also get rid of the noise as part of this so will likely post a pixel patch).
could this task be repurposed
Would you mind opening a new ticket as I'll likely have some follow up questions as I implement.