Page MenuHomePhabricator
Create Task
Maniphest T345294

Move Cloud VPS control plane alerting to alertmanager
Open, Needs TriagePublic
Actions

Description

Move from Icinga to alertmanager or replace with a better solution:

  • Galera
  • HAProxy
  • API endpoint checks
  • Instance spreadcheck
  • Nova flavor property check
  • Designate alias-dump check

Details

SubjectRepoBranchLines +/-
openstack: remove spreadcheck, absentedoperations/puppetproduction+0 -19
openstack: spreadcheck: remove in favour of server groupsoperations/puppetproduction+2 -197
P:wmcs: disable systemd icinga alertsoperations/puppetproduction+42 -56
P:openstack::galera: drop nrpe process checkoperations/puppetproduction+0 -8
openstack: Remove a bunch of Icinga checksoperations/puppetproduction+1 -393
team-wmcs: Add Galera checksoperations/alertsmaster+215 -0
team-wmcs: Add CloudLB backend status checksoperations/alertsmaster+97 -0
Move WMCS haproxy scrapes to WMCS prometheus instanceoperations/puppetproduction+40 -34
Customize query in gerrit

Related Objects
Search...

Event Timeline

taavi created this task.Aug 30 2023, 7:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 30 2023, 7:52 PM
taavi claimed this task.Aug 30 2023, 7:52 PM
gerritbot added a comment.Aug 30 2023, 7:55 PM

Change 953727 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/alerts@master] team-wmcs: Add Galera checks

https://gerrit.wikimedia.org/r/953727

gerritbot added a project: Patch-For-Review.Aug 30 2023, 7:55 PM
taavi updated the task description. (Show Details)Aug 31 2023, 9:35 AM
gerritbot added a comment.Aug 31 2023, 11:50 AM

Change 954001 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Move WMCS haproxy scrapes to WMCS prometheus instance

https://gerrit.wikimedia.org/r/954001

gerritbot added a comment.Aug 31 2023, 12:17 PM

Change 954001 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] Move WMCS haproxy scrapes to WMCS prometheus instance

https://gerrit.wikimedia.org/r/954001

taavi updated the task description. (Show Details)Aug 31 2023, 12:32 PM
gerritbot added a comment.Aug 31 2023, 1:13 PM

Change 954052 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/alerts@master] team-wmcs: Add CloudLB backend status checks

https://gerrit.wikimedia.org/r/954052

gerritbot added a comment.Aug 31 2023, 2:36 PM

Change 954052 merged by jenkins-bot:

[operations/alerts@master] team-wmcs: Add CloudLB backend status checks

https://gerrit.wikimedia.org/r/954052

gerritbot added a comment.Aug 31 2023, 4:37 PM

Change 954102 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: Remove a bunch of Icinga checks

https://gerrit.wikimedia.org/r/954102

gerritbot added a comment.Aug 31 2023, 4:53 PM

Change 953727 merged by jenkins-bot:

[operations/alerts@master] team-wmcs: Add Galera checks

https://gerrit.wikimedia.org/r/953727

gerritbot added a comment.Sep 4 2023, 10:19 AM

Change 954102 merged by Majavah:

[operations/puppet@production] openstack: Remove a bunch of Icinga checks

https://gerrit.wikimedia.org/r/954102

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2023, 10:30 AM
taavi updated the task description. (Show Details)Sep 4 2023, 11:48 AM
gerritbot added a comment.Sep 25 2023, 1:08 PM

Change 960612 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack::galera: drop nrpe process check

https://gerrit.wikimedia.org/r/960612

gerritbot added a project: Patch-For-Review.Sep 25 2023, 1:08 PM
gerritbot added a comment.Sep 25 2023, 1:16 PM

Change 960612 merged by Majavah:

[operations/puppet@production] P:openstack::galera: drop nrpe process check

https://gerrit.wikimedia.org/r/960612

Maintenance_bot removed a project: Patch-For-Review.Sep 25 2023, 1:31 PM
gerritbot added a comment.Nov 29 2023, 9:34 AM

Change 978476 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs: disable systemd icinga alerts

https://gerrit.wikimedia.org/r/978476

gerritbot added a project: Patch-For-Review.Nov 29 2023, 9:34 AM
gerritbot added a comment.Nov 29 2023, 4:03 PM

Change 978476 merged by Majavah:

[operations/puppet@production] P:wmcs: disable systemd icinga alerts

https://gerrit.wikimedia.org/r/978476

Maintenance_bot removed a project: Patch-For-Review.Nov 29 2023, 4:10 PM
gerritbot added a comment.Nov 30 2023, 12:06 PM

Change 979056 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: spreadcheck: remove in favour of server groups

https://gerrit.wikimedia.org/r/979056

gerritbot added a project: Patch-For-Review.Nov 30 2023, 12:06 PM
gerritbot added a comment.Dec 12 2023, 3:00 PM

Change 979056 merged by Majavah:

[operations/puppet@production] openstack: spreadcheck: remove in favour of server groups

https://gerrit.wikimedia.org/r/979056

Maintenance_bot removed a project: Patch-For-Review.Dec 12 2023, 3:10 PM
taavi updated the task description. (Show Details)Dec 12 2023, 3:11 PM
gerritbot added a comment.Jan 22 2024, 1:20 PM

Change 992162 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] openstack: remove spreadcheck, absented

https://gerrit.wikimedia.org/r/992162

gerritbot added a project: Patch-For-Review.Jan 22 2024, 1:20 PM
gerritbot added a comment.Jan 22 2024, 1:34 PM

Change 992162 merged by Filippo Giunchedi:

[operations/puppet@production] openstack: remove spreadcheck, absented

https://gerrit.wikimedia.org/r/992162

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2024, 2:31 PM
fnegri edited projects, added cloud-services-team (FY2023/2024-Q3-Q4); removed cloud-services-team (FY2023/2024-Q1-Q2).Feb 1 2024, 11:14 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL