Page MenuHomePhabricator
Create Task
Maniphest T272977

Self-service deployment for alerting rules
Closed, ResolvedPublic
Actions

Description

This task tracks the implementation of self-service deployment for Prometheus alerting rules, including:

  • A repository of alerting rules
  • Validation / unit testing of alerting rules
  • Deployment to production Prometheus

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -0prometheus: read alerts from /srv/alerts
operations/puppetproduction+223 -0alerts: deploy to Prometheus hosts
operations/alertsmaster+22 -0Add Blubber and Pipeline
operations/alertsmaster+134 -0Run tests for alerts
integration/configmaster+19 -2Add pipeline-based config for operations/alerts
integration/configmaster+39 -2Add tox-alerts Docker image to have 'promtool' available
integration/configmaster+4 -0Add operations/alerts tox
operations/alertsmaster+67 -0Sketch of Performance team alerts
Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.Jan 26 2021, 1:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 26 2021, 1:20 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jan 26 2021, 3:40 PM
fgiunchedi moved this task from Doing to Up next on the User-fgiunchedi board.Feb 5 2021, 1:41 PM
fgiunchedi moved this task from Inbox to In progress on the observability board.Feb 8 2021, 4:18 PM
fgiunchedi moved this task from Up next to Doing on the User-fgiunchedi board.Mar 8 2021, 8:26 AM
gerritbot added a comment.Mar 9 2021, 4:43 PM

Change 670230 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/alerts@master] Sketch of Performance team alerts

https://gerrit.wikimedia.org/r/670230

gerritbot added a project: Patch-For-Review.Mar 9 2021, 4:43 PM

Change 670231 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/alerts@master] Run tests for alerts

https://gerrit.wikimedia.org/r/670231

gerritbot added a comment.Mar 11 2021, 10:11 AM

Change 670782 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[integration/config@master] Add operations/alerts tox

https://gerrit.wikimedia.org/r/670782

gerritbot added a comment.Mar 15 2021, 8:15 AM

Change 670782 merged by jenkins-bot:
[integration/config@master] Add operations/alerts tox

https://gerrit.wikimedia.org/r/670782

gerritbot added a comment.Mar 15 2021, 8:43 AM

Change 672340 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[integration/config@master] Add tox-alerts Docker image to have 'promtool' available

https://gerrit.wikimedia.org/r/672340

gerritbot added a comment.Mar 15 2021, 10:07 AM

Change 672340 abandoned by Filippo Giunchedi:
[integration/config@master] Add tox-alerts Docker image to have 'promtool' available

Reason:
Trying the Pipeline instead

https://gerrit.wikimedia.org/r/672340

gerritbot added a comment.Mar 15 2021, 10:12 AM

Change 672359 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[integration/config@master] Add pipeline-based config for operations/alerts

https://gerrit.wikimedia.org/r/672359

gerritbot added a comment.Mar 15 2021, 10:16 AM

Change 672359 merged by jenkins-bot:
[integration/config@master] Add pipeline-based config for operations/alerts

https://gerrit.wikimedia.org/r/672359

gerritbot added a comment.Mar 15 2021, 10:44 AM

Change 672365 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/alerts@master] Add Blubber and Pipeline

https://gerrit.wikimedia.org/r/672365

gerritbot added a comment.Mar 15 2021, 1:27 PM

Change 670231 merged by Filippo Giunchedi:
[operations/alerts@master] Run tests for alerts

https://gerrit.wikimedia.org/r/670231

gerritbot added a comment.Mar 15 2021, 1:27 PM

Change 672365 merged by Filippo Giunchedi:
[operations/alerts@master] Add Blubber and Pipeline

https://gerrit.wikimedia.org/r/672365

fgiunchedi mentioned this in rOALEc490c08fa228: Add Blubber and Pipeline.Mar 15 2021, 1:27 PM
fgiunchedi mentioned this in rOALEa39fb514d7f5: Run tests for alerts.
fgiunchedi updated the task description. (Show Details)Thu, Mar 18, 8:40 AM
gerritbot added a comment.Mon, Mar 22, 2:14 PM

Change 674025 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] alerts: deploy to Prometheus hosts

https://gerrit.wikimedia.org/r/674025

gerritbot added a comment.Tue, Mar 23, 11:17 AM

Change 674025 merged by Filippo Giunchedi:
[operations/puppet@production] alerts: deploy to Prometheus hosts

https://gerrit.wikimedia.org/r/674025

gerritbot added a comment.Tue, Mar 23, 1:53 PM

Change 674321 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: read alerts from /srv/alerts

https://gerrit.wikimedia.org/r/674321

gerritbot added a comment.Wed, Mar 24, 7:38 AM

Change 674321 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: read alerts from /srv/alerts

https://gerrit.wikimedia.org/r/674321

fgiunchedi closed this task as Resolved.Thu, Mar 25, 7:40 AM
fgiunchedi claimed this task.

This is complete! Alerts will get deployed from operations/alerts to Prometheus instances

fgiunchedi updated the task description. (Show Details)Thu, Mar 25, 7:40 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL