Page MenuHomePhabricator

link Icinga checks to runbook / notes URLs
Closed, ResolvedPublic

Description

follow-up TODO item from the SRE summit in Prague:

Have wiki pages for paging Icinga alerts for playbook content how to react to them. Have them at a predictable location.


Have a wiki page or naming convention that lets responders always have a single point of persistent information for basic troubleshooting and communication
TODO: Daniel — figure out the details/make a proposal.
There is not always an icinga page for an outage, so we can’t totally rely on that for naming scheme

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+22 -7
operations/puppetproduction+2 -1
operations/puppetproduction+1 -0
operations/puppetproduction+4 -2
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+2 -0
operations/puppetproduction+3 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+70 -9
operations/puppetproduction+2 -0
operations/puppetproduction+2 -0
operations/puppetproduction+0 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+2 -1
operations/puppetproduction+6 -0
operations/puppetproduction+3 -0
operations/puppetproduction+3 -0
operations/puppetproduction+4 -1
operations/puppetproduction+2 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+2 -0
operations/puppetproduction+2 -0
operations/puppetproduction+1 -0
operations/puppetproduction+7 -0
operations/puppetproduction+2 -0
operations/puppetproduction+5 -0
operations/puppetproduction+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+3 -0
operations/puppetproduction+5 -1
operations/puppetproduction+3 -0
operations/puppetproduction+15 -0
operations/puppetproduction+9 -1
operations/puppetproduction+11 -0
operations/puppetproduction+3 -0
operations/puppetproduction+25 -2
operations/puppetproduction+4 -0
operations/puppetproduction+3 -1
operations/puppetproduction+3 -0
operations/puppetproduction+2 -0
operations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 459630 merged by Dzahn:
[operations/puppet@production] netops::check: add 3 playbook links to Icinga checks

https://gerrit.wikimedia.org/r/459630

Change 459862 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] monitoring: enable using notes_url with grafana_alert

https://gerrit.wikimedia.org/r/459862

Change 459862 merged by Dzahn:
[operations/puppet@production] monitoring: enable using notes_url with grafana_alert

https://gerrit.wikimedia.org/r/459862

Change 459641 merged by Dzahn:
[operations/puppet@production] icinga: add notes_url parameter to NRPE monitor service

https://gerrit.wikimedia.org/r/459641

fgiunchedi moved this task from In progress to Up next on the observability board.Oct 29 2018, 3:23 PM

Change 494472 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: add notes URLs to various monitoring checks, part 1

https://gerrit.wikimedia.org/r/494472

Change 494474 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/elasticsearch: add notes URLs

https://gerrit.wikimedia.org/r/494474

Change 494483 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: add notes_url for Google Safe Browsing checks

https://gerrit.wikimedia.org/r/494483

Change 494483 merged by Dzahn:
[operations/puppet@production] icinga: add notes_url for Google Safe Browsing checks

https://gerrit.wikimedia.org/r/494483

Change 494485 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/restbase/eventbus: add notes URLs

https://gerrit.wikimedia.org/r/494485

Change 494490 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/toollabs: set notes URLs for toolforge related checks

https://gerrit.wikimedia.org/r/494490

Change 494474 merged by Dzahn:
[operations/puppet@production] icinga/elasticsearch: add notes URLs

https://gerrit.wikimedia.org/r/494474

Change 494472 merged by Dzahn:
[operations/puppet@production] icinga: add notes URLs to various monitoring checks, part 1

https://gerrit.wikimedia.org/r/494472

Change 494729 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: add notes URLs to various monitoring checks, part 3

https://gerrit.wikimedia.org/r/494729

Change 494733 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] xmldumps: set notes URL and contact group to wmcs

https://gerrit.wikimedia.org/r/494733

Change 494511 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: add notes URLs to various monitoring checks, part 2

https://gerrit.wikimedia.org/r/494511

Change 494729 merged by Dzahn:
[operations/puppet@production] icinga: add notes URLs to various monitoring checks, part 3

https://gerrit.wikimedia.org/r/494729

Change 494490 merged by Dzahn:
[operations/puppet@production] icinga/toollabs: set notes URLs for toolforge related checks

https://gerrit.wikimedia.org/r/494490

Dzahn renamed this task from how to structure wiki pages for Icinga reaction play books to link Icinga checks to runbook / notes URLs .Mar 7 2019, 1:26 PM

Change 494485 merged by Dzahn:
[operations/puppet@production] icinga/restbase/eventbus: add notes URLs

https://gerrit.wikimedia.org/r/494485

Change 494733 merged by Dzahn:
[operations/puppet@production] xmldumps: set notes URL and contact group to wmcs

https://gerrit.wikimedia.org/r/494733

Change 496435 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] service: add Icinga notes URL in defined types

https://gerrit.wikimedia.org/r/496435

Change 496435 merged by Dzahn:
[operations/puppet@production] service: add Icinga notes URL in defined types

https://gerrit.wikimedia.org/r/496435

Dzahn added a comment.Mar 15 2019, 2:56 PM

A lot of changes have been merged that i did not individually link to this ticket to avoid spamming people.

There is a common topic branch, notes-urls, to list them all at once:

https://gerrit.wikimedia.org/r/q/topic:%22notes-urls%22+(status:open%20OR%20status:merged)

Change 459659 merged by Dzahn:
[operations/puppet@production] icinga: make notes_url a required parameter of monitoring::service

https://gerrit.wikimedia.org/r/459659

Change 496830 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] nrpe/icinga: make notes_url a required parameter of nrpe::monitor_service

https://gerrit.wikimedia.org/r/496830

Dzahn moved this task from Up next to In progress on the observability board.Apr 15 2019, 2:57 PM

Change 505301 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] base/monitoring: add Icinga notes_url for 'long running screens' alert

https://gerrit.wikimedia.org/r/505301

Change 505301 merged by Dzahn:
[operations/puppet@production] base/monitoring: add Icinga notes_url for 'long running screens' alert

https://gerrit.wikimedia.org/r/505301

Change 506548 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] raid: add Icinga notes_urls

https://gerrit.wikimedia.org/r/506548

Change 506551 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] installserver: add icinga runbook for tftpd down

https://gerrit.wikimedia.org/r/506551

Change 506551 merged by Dzahn:
[operations/puppet@production] installserver: add icinga runbook for tftpd down

https://gerrit.wikimedia.org/r/506551

Change 506548 merged by Dzahn:
[operations/puppet@production] raid: add Icinga notes_urls

https://gerrit.wikimedia.org/r/506548

Change 506719 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] kafka: add icinga notes URLs

https://gerrit.wikimedia.org/r/506719

Change 506550 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] base::firewall: add runbooks for check_ferm and check_conntrack

https://gerrit.wikimedia.org/r/506550

Change 506550 had a related patch set uploaded (by CDanis; owner: Dzahn):
[operations/puppet@production] base::firewall: add runbooks for check_ferm and check_conntrack

https://gerrit.wikimedia.org/r/506550

Change 506550 merged by Dzahn:
[operations/puppet@production] base::firewall: add runbooks for check_ferm and check_conntrack

https://gerrit.wikimedia.org/r/506550

Change 506719 merged by Dzahn:
[operations/puppet@production] kafka: add icinga notes URLs

https://gerrit.wikimedia.org/r/506719

Change 509478 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] haproxy: add Icinga notes_urls

https://gerrit.wikimedia.org/r/509478

Change 509478 merged by Dzahn:
[operations/puppet@production] haproxy: add Icinga notes_urls

https://gerrit.wikimedia.org/r/509478

Change 509477 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mirrrors: add Icinga notes_urls

https://gerrit.wikimedia.org/r/509477

Change 509489 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] udp2log: add Icinga notes_urls

https://gerrit.wikimedia.org/r/509489

Change 509489 merged by Dzahn:
[operations/puppet@production] udp2log: add Icinga notes_urls

https://gerrit.wikimedia.org/r/509489

Change 509532 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] burrow: add Icinga notes_url

https://gerrit.wikimedia.org/r/509532

Change 509532 merged by Dzahn:
[operations/puppet@production] burrow: add Icinga notes_url

https://gerrit.wikimedia.org/r/509532

Change 509475 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] acme_chief: add Icinga notes_url

https://gerrit.wikimedia.org/r/509475

Change 509545 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] labstore: add Icinga notes_urls

https://gerrit.wikimedia.org/r/509545

Change 509552 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mariadb: set some more Icinga notes URLs for nrpe checks

https://gerrit.wikimedia.org/r/509552

Change 509553 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] nrpe: add Icinga notes_url for systemd_unit_state check

https://gerrit.wikimedia.org/r/509553

Change 509947 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: add notes_url for bad_directory_owner check

https://gerrit.wikimedia.org/r/509947

Change 509947 merged by Dzahn:
[operations/puppet@production] icinga: add notes_url for bad_directory_owner check

https://gerrit.wikimedia.org/r/509947

Change 509553 merged by Dzahn:
[operations/puppet@production] nrpe: add Icinga notes_url for systemd_unit_state check

https://gerrit.wikimedia.org/r/509553

Change 510053 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] eventlogging: add Icinga notes URLs

https://gerrit.wikimedia.org/r/510053

Change 510054 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] statsd: add Icinga notes URL

https://gerrit.wikimedia.org/r/510054

Change 510054 merged by Dzahn:
[operations/puppet@production] statsd: add Icinga notes URL

https://gerrit.wikimedia.org/r/510054

Change 509477 merged by Dzahn:
[operations/puppet@production] mirrors: add Icinga notes_urls

https://gerrit.wikimedia.org/r/509477

Change 509552 merged by Dzahn:
[operations/puppet@production] mariadb: set some more Icinga notes URLs for nrpe checks

https://gerrit.wikimedia.org/r/509552

Change 510053 merged by Dzahn:
[operations/puppet@production] eventlogging: add Icinga notes URLs

https://gerrit.wikimedia.org/r/510053

Change 509475 merged by Dzahn:
[operations/puppet@production] acme_chief: add Icinga notes_url

https://gerrit.wikimedia.org/r/509475

Change 509545 merged by Dzahn:
[operations/puppet@production] labstore: add Icinga notes_urls

https://gerrit.wikimedia.org/r/509545

Change 511779 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] nrpe: add notes_url to monitor_systemd_unit_state

https://gerrit.wikimedia.org/r/511779

Change 511779 merged by Dzahn:
[operations/puppet@production] nrpe: add notes_url to monitor_systemd_unit_state

https://gerrit.wikimedia.org/r/511779

Change 510963 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] monitoring: add Icinga notes_url for unmerged changes check

https://gerrit.wikimedia.org/r/510963

Change 510963 merged by Dzahn:
[operations/puppet@production] monitoring: add Icinga notes_url for unmerged changes check

https://gerrit.wikimedia.org/r/510963

Change 511949 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ipmi: add Icinga notes_url

https://gerrit.wikimedia.org/r/511949

Change 520654 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/elasticsearch: fix notes_link->notes_url parameter name

https://gerrit.wikimedia.org/r/520654

Change 520654 merged by Dzahn:
[operations/puppet@production] icinga/elasticsearch: fix notes_link->notes_url parameter name

https://gerrit.wikimedia.org/r/520654

Change 520656 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga/elasticsearch: fix notes_url->dashboard_links param name

https://gerrit.wikimedia.org/r/520656

Change 520656 merged by Dzahn:
[operations/puppet@production] icinga/elasticsearch: remove notes_url param where it does not belong

https://gerrit.wikimedia.org/r/520656

Change 520747 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] monitoring::graphite_threshold: add notes_link

https://gerrit.wikimedia.org/r/520747

Change 520933 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki: add notes_urls for app servers

https://gerrit.wikimedia.org/r/520933

Change 520933 merged by Dzahn:
[operations/puppet@production] mediawiki: add notes_urls for app servers

https://gerrit.wikimedia.org/r/520933

Change 520957 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ntp/systemd: add notes_urls for timesyncd and systemd timers

https://gerrit.wikimedia.org/r/520957

Change 520957 merged by Dzahn:
[operations/puppet@production] ntp/systemd: add notes_urls for timesyncd and systemd timers

https://gerrit.wikimedia.org/r/520957

Change 520747 merged by Dzahn:
[operations/puppet@production] monitoring::graphite_threshold: add notes_link

https://gerrit.wikimedia.org/r/520747

Change 521376 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] uwsgi::app: add notes_url for services using uwsgi

https://gerrit.wikimedia.org/r/521376

Change 521376 merged by Dzahn:
[operations/puppet@production] uwsgi::app: add notes_url for services using uwsgi

https://gerrit.wikimedia.org/r/521376

Change 511949 abandoned by Dzahn:
ipmi: add Icinga notes_url

Reason:
duplicate of https://gerrit.wikimedia.org/r/c/operations/puppet/ /521401

https://gerrit.wikimedia.org/r/511949

Change 496830 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] nrpe/icinga: make notes_url a required parameter of nrpe::monitor_service

https://gerrit.wikimedia.org/r/496830

Change 496830 merged by Dzahn:
[operations/puppet@production] nrpe/icinga: make notes_url a required parameter of nrpe::monitor_service

https://gerrit.wikimedia.org/r/496830

Change 524943 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wdqs: add missing notes_url for blazegraph icinga checks

https://gerrit.wikimedia.org/r/524943

Change 524944 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] logstash: add missing notes_url links for Icinga

https://gerrit.wikimedia.org/r/524944

Change 524943 merged by Dzahn:
[operations/puppet@production] wdqs: add missing notes_url for blazegraph icinga checks

https://gerrit.wikimedia.org/r/524943

Change 524944 merged by Dzahn:
[operations/puppet@production] logstash: add missing notes_url links for Icinga

https://gerrit.wikimedia.org/r/524944

Change 524946 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mx: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524946

Change 524946 merged by Dzahn:
[operations/puppet@production] mx: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524946

Change 524947 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] acme-chief: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524947

Change 524947 merged by Dzahn:
[operations/puppet@production] acme-chief: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524947

Change 524949 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] otrs: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524949

Change 524949 merged by Dzahn:
[operations/puppet@production] otrs: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524949

Change 524950 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] etherpad: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524950

Change 524950 merged by Dzahn:
[operations/puppet@production] etherpad: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524950

Change 524951 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] netbox: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524951

Change 524951 merged by Dzahn:
[operations/puppet@production] netbox: add missing notes_url for Icinga

https://gerrit.wikimedia.org/r/524951

Dzahn closed this task as Resolved.Jul 23 2019, 12:01 AM

More patches that have been merged as part of this but did not link to this ticket to minimize notifications a bit:

https://gerrit.wikimedia.org/r/q/topic:%22notes-urls%22+(status:open%20OR%20status:merged)

This is now finally done since https://gerrit.wikimedia.org/r/c/operations/puppet/+/496830 and a few follow-ups.

Now the NRPE checks all have a notes_url and it's a required parameter. Notified the ops list.