Page MenuHomePhabricator
Create Task
Maniphest T331901

check icinga for any alerts custom to sre-collab related services (convert monitoring::service checks to prometheus)
Closed, ResolvedPublic
Actions

Description

  • modules/profile/manifests/etherpad.pp: monitoring::service { 'etherpad-lite-http':
  • modules/profile/manifests/etherpad.pp: nrpe::monitor_service { 'etherpad-lite-proc':
  • modules/profile/manifests/releases/common.pp: monitoring::service { 'https_releases':
  • modules/profile/manifests/releases/mediawiki.pp: monitoring::service { 'http_releases_jenkins':
  • modules/profile/manifests/gerrit/proxy.pp: monitoring::service { 'https':
  • modules/profile/manifests/microsites/peopleweb.pp: monitoring::service { 'https-peopleweb':
  • modules/profile/manifests/microsites/peopleweb.pp: monitoring::service { 'https-peopleweb-expiry':
  • modules/profile/manifests/microsites/static_codereview.pp: monitoring::service { 'static-codereview-http':
  • modules/profile/manifests/microsites/static_rt.pp: monitoring::service { 'static-rt-https':
  • modules/profile/manifests/vrts.pp: monitoring::service { 'smtp':
  • modules/profile/manifests/phabricator/main.pp: monitoring::service { 'smtp':
  • modules/profile/manifests/gerrit.pp: monitoring::service { 'gerrit_ssh':

Details

SubjectRepoBranchLines +/-
gerrit: replace Icinga monitoring with Prometheus, ssh port 29418operations/puppetproduction+4 -5
gerrit: follow_redirects in blackbox::http monitoringoperations/puppetproduction+1 -0
gerrit: accept http status 404 in blackbox http monitor, for nowoperations/puppetproduction+1 -1
gerrit: do not monitor the replicaoperations/puppetproduction+13 -11
gerrit: accept 200 in addition to 302 and 404 in monitoringoperations/puppetproduction+1 -1
gerrit: add Prometheus blackbox https monitoringoperations/puppetproduction+11 -0
etherpad: remove process monitoringoperations/puppetproduction+0 -7
noc: Fix alertmanager severityoperations/puppetproduction+1 -1
vrts: replace Icinga with Prometheus for SMTP monitoringoperations/puppetproduction+4 -4
noc: replace Icinga with Prometheus monitoringoperations/puppetproduction+6 -10
phabricator: replace Icinga with Prometheus for SMTP monitoringoperations/puppetproduction+4 -4
releases: rename new blackbox check for jenkins login pageoperations/puppetproduction+2 -1
releases-jenkins: replace Icinga with Prometheus monitoringoperations/puppetproduction+7 -4
etherpad: replace Icinga with Prometheus monitoringoperations/puppetproduction+13 -7
peopleweb: replace Icinga with Prometheus monitoringoperations/puppetproduction+8 -12
releases: remove Icinga monitoringoperations/puppetproduction+0 -6
alertmanager: send sre-collab alerts to -operations and -sre-collaboperations/puppetproduction+3 -2
miscweb/static_rt: replace Icinga with Prometheus monitoringoperations/puppetproduction+8 -4
Show related patches Customize query in gerrit

Related Objects

Event Timeline

Dzahn created this task.Mar 13 2023, 5:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2023, 5:08 PM
Dzahn added a comment.Mar 13 2023, 5:14 PM

The point is to go through Icinga, look at all sre-collab owned hosts and services and identify if there are any custom checks that are NOT base checks that every host has, like disk space, CPU etc.. and that have NOT already been replaced by recently added blackbox::http checks.

If there are none.. this is done..

If there are some that have been replaced by blackbox checks.. remove them.

If there is anything else special.. ask how they can be converted to alertmanager.

(sprint week related)

Dzahn added a comment.Mar 23 2023, 4:59 PM

One found and replaced with prometheus alert. This was for https://static-codereview.wikimedia.org

https://gerrit.wikimedia.org/r/c/operations/puppet/+/902226

Dzahn moved this task from Incoming to Work in Progress on the collaboration-services board.Mar 23 2023, 5:00 PM
Dzahn triaged this task as Medium priority.Mar 23 2023, 5:43 PM
Dzahn changed the task status from Open to In Progress.Mar 23 2023, 5:46 PM
Dzahn added a comment.EditedMar 23 2023, 5:52 PM

Icinga alerts to convert:

convert to blackbox::http check for collab team:

modules/profile/manifests/etherpad.pp: monitoring::service { 'etherpad-lite-http':
modules/profile/manifests/releases/common.pp: monitoring::service { 'https_releases':
modules/profile/manifests/releases/mediawiki.pp: monitoring::service { 'http_releases_jenkins':
modules/profile/manifests/gerrit/proxy.pp: monitoring::service { 'https':
modules/profile/manifests/microsites/peopleweb.pp: monitoring::service { 'https-peopleweb':
modules/profile/manifests/microsites/peopleweb.pp: monitoring::service { 'https-peopleweb-expiry':
modules/profile/manifests/microsites/static_codereview.pp: monitoring::service { 'static-codereview-http':
modules/profile/manifests/microsites/static_rt.pp: monitoring::service { 'static-rt-https':

serviceops:

modules/noc/manifests/init.pp: monitoring::service { 'https-noc':
modules/noc/manifests/init.pp: monitoring::service { 'https-noc-ssl-expiry':

not sure yet how to replace:

modules/profile/manifests/vrts.pp: monitoring::service { 'smtp':
modules/profile/manifests/phabricator/main.pp: monitoring::service { 'smtp':
modules/profile/manifests/gerrit.pp: monitoring::service { 'gerrit_ssh':

Dzahn added a project: Observability-Alerting.Mar 23 2023, 5:52 PM
gerritbot added a comment.Mar 24 2023, 7:32 PM

Change 902783 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] etherpad: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/902783

gerritbot added a project: Patch-For-Review.Mar 24 2023, 7:32 PM
gerritbot added a comment.Mar 24 2023, 7:36 PM

Change 902785 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] releases: remove Icinga monitoring

https://gerrit.wikimedia.org/r/902785

gerritbot added a comment.Mar 24 2023, 7:53 PM

Change 902788 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] releases-jenkins: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/902788

gerritbot added a comment.Mar 24 2023, 8:16 PM

Change 902799 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/902799

gerritbot added a comment.Mar 24 2023, 8:29 PM

Change 902801 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] peopleweb: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/902801

Dzahn mentioned this in T316022: Clean up check_ssl checks from puppet also covered by blackbox prober.Mar 24 2023, 8:39 PM
gerritbot added a comment.Mar 24 2023, 8:49 PM

Change 902802 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb/static_rt: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/902802

gerritbot added a comment.Mar 24 2023, 8:58 PM

Change 902802 merged by Dzahn:

[operations/puppet@production] miscweb/static_rt: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/902802

gerritbot added a comment.Mar 27 2023, 6:15 PM

Change 903318 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] alertmanager: send sre-collab alerts to -operations and -sre-collab

https://gerrit.wikimedia.org/r/903318

gerritbot added a comment.Mar 27 2023, 10:17 PM

Change 903318 merged by Dzahn:

[operations/puppet@production] alertmanager: send sre-collab alerts to -operations and -sre-collab

https://gerrit.wikimedia.org/r/903318

gerritbot added a comment.Mar 27 2023, 11:10 PM

Change 902801 merged by Dzahn:

[operations/puppet@production] peopleweb: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/902801

gerritbot added a comment.Mar 27 2023, 11:15 PM

Change 902785 merged by Dzahn:

[operations/puppet@production] releases: remove Icinga monitoring

https://gerrit.wikimedia.org/r/902785

gerritbot added a comment.Mar 27 2023, 11:18 PM

Change 902783 merged by Dzahn:

[operations/puppet@production] etherpad: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/902783

gerritbot added a comment.Mar 28 2023, 11:25 PM

Change 902788 merged by Dzahn:

[operations/puppet@production] releases-jenkins: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/902788

Dzahn updated the task description. (Show Details)Mar 28 2023, 11:37 PM
gerritbot added a comment.Mar 28 2023, 11:48 PM

Change 903801 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] noc: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/903801

Dzahn updated the task description. (Show Details)Mar 28 2023, 11:58 PM
gerritbot added a comment.Mar 29 2023, 12:06 AM

Change 903805 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] vrts: replace Icinga with Prometheus for SMTP monitoring

https://gerrit.wikimedia.org/r/903805

gerritbot added a comment.Mar 29 2023, 12:09 AM

Change 903826 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: replace Icinga with Prometheus for SMTP monitoring

https://gerrit.wikimedia.org/r/903826

gerritbot added a comment.Mar 29 2023, 12:58 PM

Change 904173 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] releases: rename new blackbox check for jenkins login page

https://gerrit.wikimedia.org/r/904173

gerritbot added a comment.Mar 29 2023, 3:52 PM

Change 904173 merged by Dzahn:

[operations/puppet@production] releases: rename new blackbox check for jenkins login page

https://gerrit.wikimedia.org/r/904173

gerritbot added a comment.Mar 30 2023, 4:05 PM

Change 903826 merged by Dzahn:

[operations/puppet@production] phabricator: replace Icinga with Prometheus for SMTP monitoring

https://gerrit.wikimedia.org/r/903826

gerritbot added a comment.Mar 30 2023, 6:08 PM

Change 903801 merged by Dzahn:

[operations/puppet@production] noc: replace Icinga with Prometheus monitoring

https://gerrit.wikimedia.org/r/903801

gerritbot added a comment.Mar 30 2023, 10:16 PM

Change 903805 merged by Dzahn:

[operations/puppet@production] vrts: replace Icinga with Prometheus for SMTP monitoring

https://gerrit.wikimedia.org/r/903805

Dzahn updated the task description. (Show Details)Mar 30 2023, 10:33 PM
gerritbot added a comment.Mar 31 2023, 9:42 PM

Change 904856 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] etherpad: remove process monitoring

https://gerrit.wikimedia.org/r/904856

gerritbot added a comment.Mar 31 2023, 10:03 PM

Change 904857 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: replace Icinga monitoring with Prometheus, ssh port 29418

https://gerrit.wikimedia.org/r/904857

gerritbot added a comment.Apr 3 2023, 10:30 AM

Change 905178 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] noc: Fix alertmanager severity

https://gerrit.wikimedia.org/r/905178

gerritbot added a comment.Apr 3 2023, 10:32 AM

Change 905178 merged by Clément Goubert:

[operations/puppet@production] noc: Fix alertmanager severity

https://gerrit.wikimedia.org/r/905178

gerritbot added a comment.Apr 3 2023, 5:54 PM

Change 904856 merged by Dzahn:

[operations/puppet@production] etherpad: remove process monitoring

https://gerrit.wikimedia.org/r/904856

Dzahn updated the task description. (Show Details)Apr 3 2023, 5:57 PM
Dzahn added a comment.Apr 6 2023, 6:10 PM

gerrit monitoring switch still in discussion/review but will be done as part of T329587

Dzahn renamed this task from check icinga for any alerts custom to sre-collab related services to check icinga for any alerts custom to sre-collab related services (convert monitoring::service checks to prometheus).Apr 6 2023, 6:20 PM
Dzahn closed this task as Resolved.Apr 6 2023, 6:28 PM
Dzahn mentioned this in T334250: sre-collab/releng: convert or remove all nrpe::monitor_service checks.

This is done but there is a continuation of it for a different class of monitoring checks. T334250

Dzahn mentioned this in T329587: http::blackbox monitoring for all https services (let all serviceops-collab alertmanager alerts create tickets).Apr 6 2023, 7:42 PM
gerritbot added a comment.Apr 20 2023, 3:49 PM

Change 904857 merged by Dzahn:

[operations/puppet@production] gerrit: replace Icinga monitoring with Prometheus, ssh port 29418

https://gerrit.wikimedia.org/r/904857

Dzahn updated the task description. (Show Details)Apr 20 2023, 4:54 PM
gerritbot added a comment.Apr 28 2023, 8:22 PM

Change 902799 merged by Dzahn:

[operations/puppet@production] gerrit: add Prometheus blackbox https monitoring

https://gerrit.wikimedia.org/r/902799

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2023, 8:30 PM
gerritbot added a comment.Apr 28 2023, 8:49 PM

Change 913262 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: follow_redirects in blackbox::http monitoring

https://gerrit.wikimedia.org/r/913262

gerritbot added a project: Patch-For-Review.Apr 28 2023, 8:49 PM
gerritbot added a comment.Apr 28 2023, 8:52 PM

Change 913262 merged by Dzahn:

[operations/puppet@production] gerrit: follow_redirects in blackbox::http monitoring

https://gerrit.wikimedia.org/r/913262

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2023, 9:10 PM
Dzahn added a comment.Apr 28 2023, 9:21 PM

https://gerrit.wikimedia.org/r/c/operations/puppet/+/913263

gerritbot added a comment.Apr 28 2023, 9:36 PM

Change 913272 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: accept http status 404 in blackbox http monitor, for now

https://gerrit.wikimedia.org/r/913272

gerritbot added a project: Patch-For-Review.Apr 28 2023, 9:36 PM
gerritbot added a comment.Apr 28 2023, 9:44 PM

Change 913272 merged by Dzahn:

[operations/puppet@production] gerrit: accept http status 404 in blackbox http monitor, for now

https://gerrit.wikimedia.org/r/913272

gerritbot added a comment.Apr 28 2023, 9:56 PM

Change 913273 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: accept 200 in addition to 302 and 404 in monitoring

https://gerrit.wikimedia.org/r/913273

gerritbot added a comment.Apr 28 2023, 9:59 PM

Change 913273 merged by Dzahn:

[operations/puppet@production] gerrit: accept 200 in addition to 302 and 404 in monitoring

https://gerrit.wikimedia.org/r/913273

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2023, 10:10 PM
gerritbot added a comment.Apr 28 2023, 10:29 PM

Change 913275 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: do not monitor the replica

https://gerrit.wikimedia.org/r/913275

gerritbot added a project: Patch-For-Review.Apr 28 2023, 10:29 PM
gerritbot added a comment.Apr 28 2023, 10:31 PM

Change 913275 merged by Dzahn:

[operations/puppet@production] gerrit: do not monitor the replica

https://gerrit.wikimedia.org/r/913275

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2023, 11:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits