Page MenuHomePhabricator
Create Task
Maniphest T309007

Migrate eventlogging check_prometheus checks to alertmanager
Closed, ResolvedPublic
Actions

Description

We have been carrying out a migration of all of our existing check_prometheus based checks from Icinga to Alertmanager.
This work has been done as part of: T293399

This ticket covers the migration of all of our (legacy) Eventlogging checks to Alertmanager.

These alerts are defined in https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/prometheus/alerts.pp#L37-L98

  • eventlogging_EventError_throughput
  • eventlogging_NavigationTiming_throughput
  • eventlogging_throughput
  • eventlogging_processors_kafka_lag

Details

SubjectRepoBranchLines +/-
Remove Eventlogging prometheus-based Icinga checksoperations/puppetproduction+0 -65
Move event logging checks from Icinga to alertmanageroperations/alertsmaster+126 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

BTullis created this task.May 23 2022, 9:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 23 2022, 9:51 AM
BTullis updated the task description. (Show Details)May 23 2022, 10:05 AM
BTullis mentioned this in T293399: Migrate the majority of the analytics cluster alerts from Icinga to AlertManager.May 23 2022, 10:09 AM
fgiunchedi subscribed.May 23 2022, 12:33 PM
odimitrijevic triaged this task as Medium priority.May 30 2022, 3:17 PM
odimitrijevic moved this task from Incoming (new tickets) to Ops Week on the Data-Engineering board.
fgiunchedi added a parent task: T288622: All Prometheus based alerts move from Icinga to alert manager exclusively.Jul 11 2022, 12:04 PM
lmata added a project: Observability-Alerting.Sep 13 2022, 1:16 AM
lmata moved this task from Inbox to Radar on the Observability-Alerting board.
gerritbot added a comment.Mar 23 2023, 12:54 PM

Change 902364 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/alerts@master] Move event logging checks from Icinga to alertmanager

https://gerrit.wikimedia.org/r/902364

gerritbot added a project: Patch-For-Review.Mar 23 2023, 12:54 PM
gerritbot added a comment.Mar 23 2023, 6:30 PM

Change 902454 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Remove Eventlogging prometheus-based Icinga checks

https://gerrit.wikimedia.org/r/902454

gerritbot added a comment.Mar 24 2023, 2:40 PM

Change 902364 merged by jenkins-bot:

[operations/alerts@master] Move event logging checks from Icinga to alertmanager

https://gerrit.wikimedia.org/r/902364

Stashbot added a comment.Mar 24 2023, 2:43 PM

Mentioned in SAL (#wikimedia-analytics) [2023-03-24T14:43:13Z] <topranks> merged alertmanager rules for eventlogging checks being migrated from Icinga T309007

gerritbot added a comment.Mar 28 2023, 8:38 PM

Change 902454 merged by Cathal Mooney:

[operations/puppet@production] Remove Eventlogging prometheus-based Icinga checks

https://gerrit.wikimedia.org/r/902454

cmooney closed this task as Resolved.Mar 28 2023, 8:39 PM
cmooney updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Mar 28 2023, 9:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits