Page MenuHomePhabricator
Create Task
Maniphest T216611

Icinga check for ircecho should check for actual activity
Closed, ResolvedPublic
Actions

Description

A restart of systemd-journald on irc.wikimedia.org caused ircecho.service to sound out further updates (https://phabricator.wikimedia.org/T216607#4968319).
While it was broken the Icinga status for kraz was all fine, the "ircecho bot process" only checks whether the udpmixecho.py process is present.

I'm not 100% sure how to best improve the check to make it detect such an error, if there are logs it could be checked whether there was any update within the last 10 seconds or so. Or given that there are channels which are very busy, the check could also join a specific channel and warn if there was no actitivity over 5-10 seconds or so.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+12 -0mw_rc_irc: add check_prometheus alert on no messages being relayed
operations/puppetproduction+33 -6profile: add prometheus job for udpmxircecho
operations/puppetproduction+10 -1mw_rc_irc: add metrics endpoint to udpmxircecho
Customize query in gerrit

Related Objects

Event Timeline

MoritzMuehlenhoff created this task.Feb 20 2019, 12:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 20 2019, 12:33 PM
MoritzMuehlenhoff renamed this task from Icnga check for ircecho should check for actual activity to Icinga check for ircecho should check for actual activity.Feb 20 2019, 12:33 PM
MoritzMuehlenhoff triaged this task as Medium priority.
CDanis added a subscriber: CDanis.Feb 20 2019, 12:52 PM

This would be an incredibly silly way to do it, but it would be very easy to write a check_prometheus invocation for outgoing network traffic from the machine:

https://grafana.wikimedia.org/d/000000377/host-overview?refresh=5m&panelId=8&fullscreen&orgId=1&var-server=kraz&var-datasource=codfw%20prometheus%2Fops&var-cluster=misc&from=1550658597855&to=1550665809997

https://grafana.wikimedia.org/d/000000377/host-overview?panelId=8&fullscreen&orgId=1&var-server=kraz&var-datasource=codfw%20prometheus%2Fops&var-cluster=misc&from=1550580548196&to=1550666948196

Probably we want something more robust. Adding an HTTP server and a few metrics to udpmxircecho.py does not look too difficult.

fgiunchedi moved this task from Inbox to Backlog on the observability board.Jul 6 2020, 12:12 PM
gerritbot added a comment.Feb 7 2021, 4:57 PM

Change 662124 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] mw_rc_irc: add metrics endpoint to udpmxircecho

https://gerrit.wikimedia.org/r/662124

gerritbot added a project: Patch-For-Review.Feb 7 2021, 4:57 PM

Change 662125 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add prometheus job for udpmxircecho

https://gerrit.wikimedia.org/r/662125

gerritbot added a comment.Feb 18 2021, 4:21 PM

Change 662124 merged by Cwhite:
[operations/puppet@production] mw_rc_irc: add metrics endpoint to udpmxircecho

https://gerrit.wikimedia.org/r/662124

Stashbot added a comment.Feb 18 2021, 4:23 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-18T16:23:35Z] <shdubsh> restart ircecho on kraz -- deploying new metrics endpoint T216611

gerritbot added a comment.Feb 18 2021, 4:36 PM

Change 662125 merged by Cwhite:
[operations/puppet@production] profile: add prometheus job for udpmxircecho

https://gerrit.wikimedia.org/r/662125

Maintenance_bot removed a project: Patch-For-Review.Feb 18 2021, 5:10 PM
gerritbot added a comment.Feb 18 2021, 5:17 PM

Change 665129 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] mw_rc_irc: add check_prometheus alert on no messages being relayed

https://gerrit.wikimedia.org/r/665129

gerritbot added a project: Patch-For-Review.Feb 18 2021, 5:17 PM
gerritbot added a comment.Feb 23 2021, 8:54 PM

Change 665129 merged by Cwhite:
[operations/puppet@production] mw_rc_irc: add check_prometheus alert on no messages being relayed

https://gerrit.wikimedia.org/r/665129

Maintenance_bot removed a project: Patch-For-Review.Feb 23 2021, 9:12 PM
colewhite closed this task as Resolved.Feb 23 2021, 9:17 PM
colewhite claimed this task.
colewhite added a subscriber: colewhite.

Updated monitoring has been deployed.

MoritzMuehlenhoff added a comment.Mar 16 2021, 10:30 AM

This has been flapping in Icinga, e.g. for today:

[11:13] <icinga-wm> PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job={ircd,netbox_device_statistics} site=codfw https://wikitech.wikimedia.org/wiki
[11:22] <icinga-wm> RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[11:24] <icinga-wm> PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job=ircd site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets
[11:25] <icinga-wm> RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets

The irc service itself was working just fine (had a connection #de-wikipedia open which was streaming edits just fine)

MoritzMuehlenhoff reopened this task as Open.Mar 16 2021, 10:31 AM
colewhite closed this task as Resolved.Mar 16 2021, 3:40 PM

Digging a bit into this found that there is some breakdown in communication between prometheus-ircd-exporter and ircd on the new irc2001 host: T224579

ircecho does not appear affected.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL