Page MenuHomePhabricator
Create Task
Maniphest T404677

Add alert for Pixel 2 that fires when Pixel is down.
Open, Needs TriagePublic1 Estimated Story Points
Actions

Description

Alert when Pixel 2 do not produce any data to the data source. See T403057

AC:

  • Create the alert in Grafana
  • Create a task for creating a runbook on have to handle that alert

Related Objects

Event Timeline

Peter created this task.Sep 16 2025, 6:55 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 16 2025, 6:55 AM
Peter mentioned this in T403057: Design observability and monitoring for Pixel 2.Sep 16 2025, 6:55 AM
Peter moved this task from Backlog to Ready to Refine on the Test-Platform board.Oct 7 2025, 1:49 PM
Peter claimed this task.Oct 7 2025, 8:14 PM
Peter set the point value for this task to 1.
Peter moved this task from Ready to Refine to dek kvin (Current Sprint) on the Test-Platform board.
Peter edited projects, added Test-Platform (dek kvin (Current Sprint)); removed Test-Platform.
Peter moved this task from Backlog to In Progress on the Test-Platform (dek kvin (Current Sprint)) board.Oct 8 2025, 5:29 AM
Peter added a comment.Oct 8 2025, 7:14 AM

I couldn't get the email alerts to work and it seems that either it has been disabling or a configuration is missing T406689

Peter added a comment.Oct 8 2025, 7:53 AM

I've setup an alert now that fires if we have two hours of no new data. I think that's the simplest way to do it. I configure it to send emails for now (to me and to our reporting Slack channel).

Peter added a comment.Oct 15 2025, 3:52 PM

There's another supported way now to send Slack alerts: https://wikitech.wikimedia.org/wiki/Alertmanager#Sending_alerts_to_Slack

Peter removed Peter as the assignee of this task.Oct 31 2025, 7:10 AM
Peter moved this task from dek kvin (Current Sprint) to Backlog on the Test-Platform board.
Peter edited projects, added Test-Platform; removed Test-Platform (dek kvin (Current Sprint)).

This task is blocked on the observability team for now.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits