Page MenuHomePhabricator
Create Task
Maniphest T345190

Update RL alerts from performance-team-alerts@ to mediawiki-platform-team@
Closed, ResolvedPublic
Actions

Description

  • Learn a bit about how and where "team" contacts are defined in AlertManager.
  • How individual alerts in Grafana specify which team to notify.
  • Create a contact for MediaWiki Platform Team and its email address / IRC bots channel.
  • Update all RL alerts to notify this team instead.

Example:
https://grafana-rw.wikimedia.org/alerting/dvAFSjJ4k/edit?returnTo=%2Fd%2F000000402%2Fresourceloader-alerts%3ForgId%3D1%26editPanel%3D16%26tab%3Dalert

Details

SubjectRepoBranchLines +/-
search-grafana-dashboards: add support for searching alert metadataoperations/softwaremaster+32 -6
Update RL alerts from performance-team-alerts@ to mediawiki-platform-team@operations/puppetproduction+10 -0
Customize query in gerrit

Related Objects

Event Timeline

Krinkle created this task.Aug 29 2023, 5:02 PM
Restricted Application added a project: MediaWiki-Platform-Team. · View Herald TranscriptAug 29 2023, 5:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
larissagaulia added a project: MediaWiki-Engineering-Group-onboarding.Aug 30 2023, 12:31 PM
larissagaulia moved this task from Inbox, needs triage to Soon on the MediaWiki-Platform-Team board.Sep 4 2023, 11:03 AM
Krinkle assigned this task to Hokwelum.Sep 5 2023, 4:53 PM

Let's pair on this once you're back!

Krinkle triaged this task as Medium priority.Sep 5 2023, 4:53 PM
pmiazga moved this task from Soon to Current Sprint on the MediaWiki-Platform-Team board.Sep 11 2023, 1:51 PM
Krinkle updated the task description. (Show Details)Sep 12 2023, 7:04 PM
fgiunchedi subscribed.Sep 13 2023, 10:03 AM

@Hokwelum you can find alertmanager onboard documentation for new teams at https://wikitech.wikimedia.org/wiki/Alertmanager . Please add me or other members of observability to the gerrit code reviews and we'll review/merge them for you! Please reach out too if you need further assistance and/or have questions

gerritbot added a comment.Sep 14 2023, 8:11 AM

Change 957664 had a related patch set uploaded (by Hokwelum; author: Hokwelum):

[operations/puppet@production] Update RL alerts from performance-team-alerts@ to mediawiki-platform-team@

https://gerrit.wikimedia.org/r/957664

gerritbot added a project: Patch-For-Review.Sep 14 2023, 8:11 AM
gerritbot added a comment.Sep 15 2023, 1:44 PM

Change 957664 merged by Filippo Giunchedi:

[operations/puppet@production] Update RL alerts from performance-team-alerts@ to mediawiki-platform-team@

https://gerrit.wikimedia.org/r/957664

Maintenance_bot removed a project: Patch-For-Review.Sep 15 2023, 2:10 PM
Krinkle moved this task from Inbox to Accepted Enhancement on the MediaWiki-ResourceLoader board.Sep 20 2023, 9:55 PM
gerritbot added a comment.Sep 21 2023, 1:04 AM

Change 959366 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/software@master] search-grafana-dashboards: add support for searching alert metadata

https://gerrit.wikimedia.org/r/959366

gerritbot added a project: Patch-For-Review.Sep 21 2023, 1:04 AM
Krinkle mentioned this in T347221: Grafana alerts unreliable since 29 August 2023 due to DatasourceError.Sep 23 2023, 2:05 AM
gerritbot added a comment.Oct 2 2023, 7:36 AM

Change 959366 merged by Filippo Giunchedi:

[operations/software@master] search-grafana-dashboards: add support for searching alert metadata

https://gerrit.wikimedia.org/r/959366

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2023, 8:11 AM
DAlangi_WMF subscribed.Oct 5 2023, 11:30 AM

Is there anything left to do here @Krinkle or @Hokwelum? Can we resolve this?

Hokwelum closed this task as Resolved.Oct 5 2023, 5:15 PM

Done!

Krinkle mentioned this in T349159: Transfer Arc Lamp alerts (Grafana and cron errors_mailto) to SRE O11y.Oct 18 2023, 2:58 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL