Page MenuHomePhabricator
Create Task
Maniphest T281359

Onboard teams with Grafana alerts to AlertManager
Closed, ResolvedPublic
Actions

Description

There are a few teams with Grafana alerts in Icinga at the moment, we should onboard these teams to AlertManager and move the Grafana alerts off Icinga.

The following checks currently use Grafana:

monitor/ores.pp:    monitoring::grafana_alert { 'ores':
monitor/ores.pp:    monitoring::grafana_alert { 'ores-extension':
monitor/reading_web.pp:    monitoring::grafana_alert { 'reading-web-page-previews':
monitor/reading_web.pp:    monitoring::grafana_alert { 'wikimedia-client-errors-alerts':
monitor/services.pp:    monitoring::grafana_alert { 'restbase-legacy':
monitor/services.pp:    monitoring::grafana_alert { 'restbase':
monitor/services.pp:    monitoring::grafana_alert { 'change-propagation':
monitor/services.pp:    monitoring::grafana_alert { 'jobqueue-eventbus':
monitor/traffic.pp:    monitoring::grafana_alert { 'varnish-http-requests':
monitor/traffic.pp:    monitoring::grafana_alert { 'ping-offload':
monitor/traffic.pp:    monitoring::grafana_alert { 'rpki':
monitor/wikidata.pp:    monitoring::grafana_alert { 'wikidata-alerts':
ORES
Reading Web
  • reading-web-page-previews: dashboard. Median TTP alert
  • wikimedia-client-errors-alerts: dashboard. Client errors alert
Services
  • restbase-legacy: dashboard. No alerts AFAICS
  • restbase: dashboard. No alerts AFAICS
  • change-propagation dashboard. No alerts AFAICS
  • jobqueue-eventbus: dashboard not found
Traffic
  • varnish-http-requests: dashboard. Traffic drop alert
  • ping-offload: dashboard. Input IP error rates
  • rpki: dashboard. Misc RPKI / routinator alerts.
Wikidata
  • wikidata-alerts: dashboard. Misc wikidata alerts

Details

SubjectRepoBranchLines +/-
icinga: remove check_grafana_alertoperations/puppetproduction+0 -128
icinga: remove reading-web Grafana checksoperations/puppetproduction+0 -11
alertmanager: route readers web team alertsoperations/puppetproduction+7 -0
Remove Grafana alerts for ORESoperations/puppetproduction+0 -18
icinga: remove services Grafana alertsoperations/puppetproduction+0 -29
icinga: update reading web Grafana alertsoperations/puppetproduction+1 -9
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Apr 28 2021, 12:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 28 2021, 12:42 PM
fgiunchedi added projects: observability, User-fgiunchedi.Apr 28 2021, 12:43 PM
fgiunchedi updated the task description. (Show Details)Apr 28 2021, 12:47 PM
fgiunchedi updated the task description. (Show Details)Apr 28 2021, 2:37 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Apr 29 2021, 9:29 AM
lmata moved this task from Inbox to In progress on the observability board.May 3 2021, 3:36 PM
Gehel added a project: Wikidata-Query-Service.May 26 2021, 11:17 AM
Maintenance_bot added a project: Wikidata.May 26 2021, 11:45 AM
MPhamWMF triaged this task as High priority.Jun 7 2021, 3:43 PM
MPhamWMF moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.
TJones renamed this task from Onboard teams with Grafana alerts to AM to Onboard teams with Grafana alerts to AlertManager.Jun 7 2021, 3:43 PM
TJones updated the task description. (Show Details)
gerritbot added a comment.Jun 8 2021, 9:22 AM

Change 698735 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] icinga: update reading web Grafana alerts

https://gerrit.wikimedia.org/r/698735

gerritbot added a project: Patch-For-Review.Jun 8 2021, 9:22 AM
gerritbot added a comment.Jun 8 2021, 12:10 PM

Change 698735 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: update reading web Grafana alerts

https://gerrit.wikimedia.org/r/698735

Maintenance_bot removed a project: Patch-For-Review.Jun 8 2021, 1:10 PM
lmata edited projects, added SRE Observability (FY2021/2022-Q1); removed observability.Jul 12 2021, 2:20 AM
lmata moved this task from Inbox to In progress on the SRE Observability (FY2021/2022-Q1) board.
fgiunchedi updated the task description. (Show Details)Jul 26 2021, 10:15 AM
gerritbot added a comment.Jul 27 2021, 10:18 PM

Change 708369 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/puppet@production] alertmanager: route readers web team alerts

https://gerrit.wikimedia.org/r/708369

gerritbot added a project: Patch-For-Review.Jul 27 2021, 10:18 PM
gerritbot added a comment.Jul 28 2021, 8:20 AM

Change 708369 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: route readers web team alerts

https://gerrit.wikimedia.org/r/708369

gerritbot added a comment.Jul 28 2021, 8:33 AM

Change 708476 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] icinga: remove services Grafana alerts

https://gerrit.wikimedia.org/r/708476

gerritbot added a comment.Jul 28 2021, 2:26 PM

Change 708476 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: remove services Grafana alerts

https://gerrit.wikimedia.org/r/708476

fgiunchedi updated the task description. (Show Details)Jul 28 2021, 2:28 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 28 2021, 3:10 PM
gerritbot added a comment.Jul 29 2021, 7:49 AM

Change 708719 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Remove Grafana alerts for ORES

https://gerrit.wikimedia.org/r/708719

gerritbot added a project: Patch-For-Review.Jul 29 2021, 7:49 AM
gerritbot added a comment.Jul 29 2021, 8:00 AM

Change 708719 merged by Elukey:

[operations/puppet@production] Remove Grafana alerts for ORES

https://gerrit.wikimedia.org/r/708719

fgiunchedi updated the task description. (Show Details)Jul 29 2021, 8:27 AM
Addshore mentioned this in T287741: Convert wikidata-alerts grafana dashboard to AlertManager.Jul 30 2021, 9:14 AM
fgiunchedi closed subtask T282806: Port traffic/netops grafana alerts to AlertManager as Resolved.Aug 2 2021, 8:29 AM
fgiunchedi added a subtask: T287741: Convert wikidata-alerts grafana dashboard to AlertManager.Aug 2 2021, 9:08 AM
fgiunchedi added a parent task: T288622: All Prometheus based alerts move from Icinga to alert manager exclusively.Aug 12 2021, 9:00 AM
lmata added a project: Goal.Aug 18 2021, 2:10 PM
lmata claimed this task.Aug 18 2021, 4:41 PM
gerritbot added a comment.Aug 20 2021, 3:52 PM

Change 714067 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] icinga: remove reading-web Grafana checks

https://gerrit.wikimedia.org/r/714067

gerritbot added a comment.Aug 23 2021, 7:50 AM

Change 714067 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: remove reading-web Grafana checks

https://gerrit.wikimedia.org/r/714067

fgiunchedi updated the task description. (Show Details)Aug 23 2021, 8:02 AM
MPhamWMF removed a project: Wikidata-Query-Service.Aug 23 2021, 3:53 PM
Ladsgroup updated the task description. (Show Details)Aug 30 2021, 4:49 PM
Addshore closed subtask T287741: Convert wikidata-alerts grafana dashboard to AlertManager as Resolved.Aug 31 2021, 5:09 PM
lmata moved this task from In progress to Done on the SRE Observability (FY2021/2022-Q1) board.Aug 31 2021, 9:57 PM
fgiunchedi moved this task from Done to In progress on the SRE Observability (FY2021/2022-Q1) board.Sep 1 2021, 8:19 AM
gerritbot added a comment.Sep 6 2021, 12:38 PM

Change 719107 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] icinga: remove check_grafana_alert

https://gerrit.wikimedia.org/r/719107

gerritbot added a comment.Sep 8 2021, 6:38 AM

Change 719107 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: remove check_grafana_alert

https://gerrit.wikimedia.org/r/719107

fgiunchedi moved this task from In progress to Done on the SRE Observability (FY2021/2022-Q1) board.Sep 8 2021, 6:39 AM
fgiunchedi closed this task as Resolved.Sep 28 2021, 11:48 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL