Yesterday @Andrew was doing maintenance on wikitech-static and put the check in icinga in maint but shinken still alerted. let's do the following:
https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=1&host=wikitech-static.wikimedia.org
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • chasemp | T178405 create a wmcs alerting group in icinga and review alerting | |||
| Resolved | • aborrero | T189584 wikitech static monitoring changes |
Change 419236 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] icinga: update wikitech-static check contacts
Change 419237 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] shinken: labs: delete wikitech-static check
Change 419237 abandoned by Arturo Borrero Gonzalez:
shinken: labs: delete wikitech-static check
Reason:
See https://gerrit.wikimedia.org/r/c/419236/?
Change 419236 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] icinga: refresh wikitech-static monitoring and alerting
Icinga seems to accept our new config:
aborrero@einsteinium:~$ sudo puppet agent -t -v Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for einsteinium.wikimedia.org Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files Info: Applying configuration version '1520966261' Notice: /Stage[main]/Icinga::Monitor::Certs/Monitoring::Host[wikitech-static.wikimedia.org]/Nagios_host[wikitech-static.wikimedia.org]/contact_groups: contact_groups changed 'wikitech-static' to 'wmcs-bots,admins' Info: Computing checksum on file /etc/nagios/nagios_host.cfg Info: Monitoring::Host[wikitech-static.wikimedia.org]: Scheduling refresh of Service[icinga] Notice: /Stage[main]/Icinga::Monitor::Certs/Monitoring::Service[https_wikitech-static]/Nagios_service[einsteinium https_wikitech-static]/contact_groups: contact_groups changed 'wikitech-static' to 'wmcs-bots,admins' Info: Computing checksum on file /etc/nagios/nagios_service.cfg Info: Monitoring::Service[https_wikitech-static]: Scheduling refresh of Service[icinga] Notice: /Stage[main]/Icinga::Monitor::Etcd_mw_config/Systemd::Service[update-etcd-mw-config-lastindex]/Service[update-etcd-mw-config-lastindex]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Icinga::Monitor::Etcd_mw_config/Systemd::Service[update-etcd-mw-config-lastindex]/Service[update-etcd-mw-config-lastindex]: Unscheduling refresh on Service[update-etcd-mw-config-lastindex] Notice: /Stage[main]/Icinga::Web/Letsencrypt::Cert::Integrated[icinga]/Exec[acme-setup-acme-icinga]/returns: executed successfully Notice: /Stage[main]/Icinga/Service[icinga]: Triggered 'refresh' from 2 events Notice: Applied catalog in 40.55 seconds
and:
aborrero@einsteinium:~$ sudo tailf /var/log/icinga/icinga.log [1520966219] SERVICE ALERT: mw2144;Nginx local proxy to apache;OK;SOFT;2;HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.199 second response time [1520966229] SERVICE ALERT: cp3032;IPsec;OK;SOFT;2;Strongswan OK - 56 ESP OK [1520966229] SERVICE ALERT: api.svc.codfw.wmnet;LVS HTTP IPv4;OK;SOFT;2;HTTP OK: HTTP/1.1 200 OK - 25599 bytes in 0.405 second response time [1520966279] SERVICE ALERT: gerrit2001;Check systemd state;CRITICAL;SOFT;1;CRITICAL - degraded: The system is operational but one or more units failed. [1520966339] SERVICE ALERT: gerrit2001;Check systemd state;CRITICAL;SOFT;2;CRITICAL - degraded: The system is operational but one or more units failed. [1520966342] Caught SIGHUP, restarting... [1520966342] Icinga 1.11.6 starting... (PID=38924) [1520966342] Local time is Tue Mar 13 18:39:02 UTC 2018 [1520966342] LOG VERSION: 2.0 [1520966352] Event loop started...
The new icinga test ('Wikitech-static main page has content') has been alerting critical for a while. The actual site seems to be up and working.
icinga error message is 140555763406480:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:827: which isn't very useful.
I can jump tomorrow into einsteinium and run the check by hand to see if more information can be obtained.
Change 419494 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wikitech-static: monitor using check_https_url_at_address_for_string
Change 419494 merged by Andrew Bogott:
[operations/puppet@production] wikitech-static: monitor using check_https_url_at_address_for_string
Hey @Andrew sorry for not handling this in a timely manner. Is this solved now? I don't see anything weird in icinga.