Page MenuHomePhabricator
Create Task
Maniphest T282806

Port traffic/netops grafana alerts to AlertManager
Closed, ResolvedPublic
Actions

Description

From parent task:

monitor/traffic.pp:    monitoring::grafana_alert { 'varnish-http-requests':
monitor/traffic.pp:    monitoring::grafana_alert { 'ping-offload':
monitor/traffic.pp:    monitoring::grafana_alert { 'rpki':

See also: https://wikitech.wikimedia.org/wiki/Alertmanager

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

BBlack created this task.May 13 2021, 6:10 PM
BBlack updated the task description. (Show Details)May 13 2021, 6:13 PM
lmata subscribed.May 13 2021, 6:14 PM
Maintenance_bot added a project: SRE.May 13 2021, 6:45 PM
ssingh subscribed.May 19 2021, 2:48 PM
lmata moved this task from Inbox to Radar on the observability board.May 19 2021, 5:47 PM

moving to radar for tracking :-)

fgiunchedi moved this task from Backlog to Radar on the User-fgiunchedi board.May 20 2021, 2:57 PM
fgiunchedi moved this task from Radar to Doing on the User-fgiunchedi board.
• Marostegui removed a project: SRE.May 24 2021, 11:41 AM
Maintenance_bot added a project: SRE.May 24 2021, 11:45 AM
• Marostegui removed a project: SRE.May 25 2021, 4:52 AM
• ema subscribed.May 25 2021, 1:50 PM
• ema triaged this task as Medium priority.May 26 2021, 2:42 PM
gerritbot added a comment.May 26 2021, 3:35 PM

Change 695367 had a related patch set uploaded (by Ema; author: Ema):

[operations/puppet@production] alertmanager: route Traffic team alerts

https://gerrit.wikimedia.org/r/695367

gerritbot added a project: Patch-For-Review.May 26 2021, 3:35 PM
gerritbot added a comment.May 27 2021, 8:57 AM

Change 695367 merged by Ema:

[operations/puppet@production] alertmanager: route Traffic team alerts

https://gerrit.wikimedia.org/r/695367

Maintenance_bot removed a project: Patch-For-Review.May 27 2021, 9:11 AM
• ema added a comment.May 27 2021, 9:38 AM

I've added a always-firing test alert on Grafana with the following tags: team: traffic, severity: critical. Shortly after I did so, we received an alert both via email and IRC, confirming that alertmanager routing works as expected.

09:34 -!- jinxer-wm [~jinxer-wm@user/jinxer-wm] has joined #wikimedia-traffic
09:34 < jinxer-wm> (EmaTestingAlertManager) firing: EmaTestingAlertManager - https://alerts.wikimedia.org
gerritbot added a comment.May 27 2021, 12:37 PM

Change 696384 had a related patch set uploaded (by Ema; author: Ema):

[operations/puppet@production] icinga: remove Grafana alerts for Traffic/Netops

https://gerrit.wikimedia.org/r/696384

gerritbot added a project: Patch-For-Review.May 27 2021, 12:37 PM
• ema renamed this task from Port traffic/netops grafana alerts to AM to Port traffic/netops grafana alerts to AlertManager.May 27 2021, 12:37 PM
• ema added a comment.May 27 2021, 2:05 PM

OK so it turns out that defining the alerts in Grafana is possible but not recommended, and the right thing to do is adding them to the operations/alert repo instead. My bad!

gerritbot added a comment.May 27 2021, 2:24 PM

Change 696468 had a related patch set uploaded (by Ema; author: Ema):

[operations/alerts@master] Traffic team alerts

https://gerrit.wikimedia.org/r/696468

gerritbot added a comment.Jun 1 2021, 2:31 PM

Change 696468 merged by Ema:

[operations/alerts@master] Traffic team alerts

https://gerrit.wikimedia.org/r/696468

• ema mentioned this in rOALEbfbe9c5290a6: Traffic team alerts.Jun 1 2021, 2:31 PM
gerritbot added a comment.Jun 2 2021, 8:12 AM

Change 697710 had a related patch set uploaded (by Ema; author: Ema):

[operations/alerts@master] Netops team alert: ping offload

https://gerrit.wikimedia.org/r/697710

gerritbot added a comment.Jun 2 2021, 9:43 AM

Change 697721 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alertmanager: attach runbook/dashboard URLs to IRC messages

https://gerrit.wikimedia.org/r/697721

gerritbot added a comment.Jun 2 2021, 9:43 AM

Change 697722 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alertmanager: add a sample JSON alert and instruction on how to test IRC format changes

https://gerrit.wikimedia.org/r/697722

gerritbot added a comment.Jun 2 2021, 12:15 PM

Change 697737 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alerts: reload prometheus instances after deploy

https://gerrit.wikimedia.org/r/697737

gerritbot added a comment.Jun 3 2021, 9:41 AM

Change 697721 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: attach runbook/dashboard URLs to IRC messages

https://gerrit.wikimedia.org/r/697721

gerritbot added a comment.Jun 3 2021, 9:41 AM

Change 697722 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: add a sample alert and test instructions

https://gerrit.wikimedia.org/r/697722

gerritbot added a comment.Jun 3 2021, 9:41 AM

Change 697737 merged by Filippo Giunchedi:

[operations/puppet@production] alerts: reload prometheus instances after deploy

https://gerrit.wikimedia.org/r/697737

gerritbot added a comment.Jun 3 2021, 10:18 AM

Change 697924 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alertmanager: highlight 'instance' label in alerts dashboard

https://gerrit.wikimedia.org/r/697924

gerritbot added a comment.Jun 4 2021, 8:24 AM

Change 697924 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: highlight 'instance' label in alerts dashboard

https://gerrit.wikimedia.org/r/697924

• ema moved this task from Backlog to Caching on the Traffic board.Jun 4 2021, 10:23 AM
gerritbot added a comment.Jun 7 2021, 7:53 AM

Change 698459 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] alertmanager: print link separators on IRC when needed

https://gerrit.wikimedia.org/r/698459

gerritbot added a comment.Jun 7 2021, 12:59 PM

Change 698491 had a related patch set uploaded (by Ema; author: Ema):

[operations/puppet@production] alertmanager: define IRC and page routes for sre team

https://gerrit.wikimedia.org/r/698491

gerritbot added a comment.Jun 7 2021, 1:40 PM

Change 698491 merged by Ema:

[operations/puppet@production] alertmanager: define IRC and page routes for sre team

https://gerrit.wikimedia.org/r/698491

gerritbot added a comment.Jun 7 2021, 2:28 PM

Change 697710 merged by Ema:

[operations/alerts@master] Netops team alert: ping offload

https://gerrit.wikimedia.org/r/697710

• ema mentioned this in rOALE5f3a75531189: Netops team alert: ping offload.Jun 7 2021, 2:29 PM
gerritbot added a comment.Jun 7 2021, 2:43 PM

Change 698548 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] pipeline: use bullseye to get newer prometheus

https://gerrit.wikimedia.org/r/698548

gerritbot added a comment.Jun 7 2021, 2:56 PM

Change 698548 merged by Filippo Giunchedi:

[operations/alerts@master] pipeline: use bullseye to get newer prometheus

https://gerrit.wikimedia.org/r/698548

fgiunchedi mentioned this in rOALE986baa0ba626: pipeline: use bullseye to get newer prometheus.Jun 7 2021, 2:56 PM
gerritbot added a comment.Jun 8 2021, 9:49 AM

Change 698459 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: print link separators on IRC when needed

https://gerrit.wikimedia.org/r/698459

fgiunchedi mentioned this in T284810: Add ability to select which Prometheus instance an alert file belongs to.Jun 11 2021, 9:20 AM
gerritbot added a comment.Jun 21 2021, 3:29 PM

Change 700649 had a related patch set uploaded (by Ayounsi; author: XioNoX):

[operations/alerts@master] Move RPKI alerts to Prometheus/AM

https://gerrit.wikimedia.org/r/700649

gerritbot added a comment.Jul 1 2021, 3:53 PM

Change 700649 merged by Ayounsi:

[operations/alerts@master] Move RPKI alerts to Prometheus/AM

https://gerrit.wikimedia.org/r/700649

ayounsi mentioned this in rOALEffa3a2fbc5c7: Move RPKI alerts to Prometheus/AM.Jul 1 2021, 3:53 PM
gerritbot added a comment.Jul 1 2021, 3:58 PM

Change 702688 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Remove old RPKI Grafana alerts

https://gerrit.wikimedia.org/r/702688

gerritbot added a comment.Jul 1 2021, 4:02 PM

Change 702688 merged by Ayounsi:

[operations/puppet@production] Remove old RPKI Grafana alerts

https://gerrit.wikimedia.org/r/702688

gerritbot added a comment.Jul 26 2021, 10:14 AM

Change 708081 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] icinga: remove grafana alerts for Traffic, moved to alertmanager

https://gerrit.wikimedia.org/r/708081

gerritbot added a comment.Aug 2 2021, 8:27 AM

Change 708081 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: remove grafana alerts for Traffic, moved to alertmanager

https://gerrit.wikimedia.org/r/708081

fgiunchedi closed this task as Resolved.Aug 2 2021, 8:29 AM
fgiunchedi claimed this task.

This is complete! Thanks all

• ema mentioned this in T291148: VarnishTrafficDrop alert false positives due to DCs depooled.Sep 16 2021, 7:09 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL