Page MenuHomePhabricator
Create Task
Maniphest T287741

Convert wikidata-alerts grafana dashboard to AlertManager
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Prompted by T281359: Onboard teams with Grafana alerts to AlertManager

Dashboard: https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts?orgId=1&refresh=5s
Docs: https://wikitech.wikimedia.org/wiki/Alertmanager

For anything either not covered or missing in the docs we are more than happy to assist with the work

using grafana for alerts is still supported, but alerts should be directed to AM, or alternatively alerts as prometheus rules deployed to alerts.git

Acceptance criteria 🏕️🌟

  • Grafana directs alerts for wikidata-alerts dashboard to alertmanager

Details

SubjectRepoBranchLines +/-
icinga: Drop grafana alertsoperations/puppetproduction+0 -6
alertmanager: Add Wikidata team to alert manageroperations/puppetproduction+7 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Addshore created this task.Jul 30 2021, 9:14 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 30 2021, 9:14 AM
Maintenance_bot added a project: Wikidata.Jul 30 2021, 9:45 AM
fgiunchedi added a parent task: T281359: Onboard teams with Grafana alerts to AlertManager.Aug 2 2021, 9:08 AM
lmata moved this task from Inbox to Backlog on the Observability-Alerting board.Aug 10 2021, 3:18 PM
Addshore moved this task from wdwb-tech (small tasks) to Inbox on the [DEPRECATED] wdwb-tech board.Aug 18 2021, 9:40 AM
Addshore edited projects, added [DEPRECATED] wdwb-tech; removed [DEPRECATED] wdwb-tech (wdwb-tech (small tasks)).
WMDE-leszek subscribed.Aug 20 2021, 3:18 PM
Addshore moved this task from Inbox to To Prioritize on the [DEPRECATED] wdwb-tech board.Aug 24 2021, 9:10 AM
Addshore moved this task from To Prioritize to Triaged High (100+) on the [DEPRECATED] wdwb-tech board.Aug 24 2021, 10:38 AM
Addshore added a project: Wikidata-Campsite.Aug 24 2021, 10:41 AM
Addshore moved this task from Incoming to Prioritized Wikidata Tech Backlog (prioritised from top to bottom) on the Wikidata-Campsite board.
Addshore updated the task description. (Show Details)Aug 25 2021, 8:34 AM
Addshore updated the task description. (Show Details)Aug 25 2021, 10:11 AM
Addshore updated the task description. (Show Details)Aug 25 2021, 10:17 AM
Addshore set the point value for this task to 8.Aug 25 2021, 10:20 AM
Addshore moved this task from Prioritized Wikidata Tech Backlog (prioritised from top to bottom) to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.Aug 25 2021, 10:25 AM
Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Ladsgroup claimed this task.Aug 30 2021, 12:51 PM
Ladsgroup moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 30 2021, 12:51 PM
gerritbot added a comment.Aug 30 2021, 2:06 PM

Change 715505 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] alertmanager: Add Wikidata team to alert manager

https://gerrit.wikimedia.org/r/715505

gerritbot added a project: Patch-For-Review.Aug 30 2021, 2:06 PM
gerritbot added a comment.Aug 30 2021, 2:19 PM

Change 715505 merged by Filippo Giunchedi:

[operations/puppet@production] alertmanager: Add Wikidata team to alert manager

https://gerrit.wikimedia.org/r/715505

Maintenance_bot removed a project: Patch-For-Review.Aug 30 2021, 3:10 PM
Michael subscribed.Aug 30 2021, 3:31 PM
gerritbot added a comment.Aug 30 2021, 3:42 PM

Change 715543 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] icinga: Drop grafana alerts

https://gerrit.wikimedia.org/r/715543

gerritbot added a project: Patch-For-Review.Aug 30 2021, 3:42 PM
gerritbot added a comment.Aug 30 2021, 3:50 PM

Change 715543 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: Drop grafana alerts

https://gerrit.wikimedia.org/r/715543

Ladsgroup moved this task from Doing to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 30 2021, 4:01 PM

The grafana part is done, there are some other wikidata alerts that need migrating to AM which I will do separately on the parent ticket.

Maintenance_bot removed a project: Patch-For-Review.Aug 30 2021, 4:10 PM
Addshore mentioned this in T287587: wikibase registry didn't send an email when it ran out of disk space.Aug 31 2021, 10:50 AM
Addshore moved this task from Triaged High (100+) to Active on the [DEPRECATED] wdwb-tech board.Aug 31 2021, 11:21 AM
fgiunchedi mentioned this in T290080: Move wikidata lag checks off Icinga.Aug 31 2021, 2:53 PM
Addshore closed this task as Resolved.Aug 31 2021, 5:09 PM

Looks like the grafana ones are all done.
@Ladsgroup also created T290080: Move wikidata lag checks off Icinga as there are some puppet metrics that we could move over.
While discussing on mattermost it seems that these are only for dispatch lag, which is already sent to graphite.
SO we can probably just ensure we have matching alerts for these, and nuke the duplicated puppet alerts?

lmata moved this task from Backlog to Done on the Observability-Alerting board.Jan 16 2023, 5:57 PM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Jan 16 2023, 6:15 PM
Maintenance_bot moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL