In the process of deprecating/removing icinga we have a few scripts invoked via nrpe (i.e. locally on the host, as opposed on the icinga host). It would be convenient to at least evaluate if we can ease the icinga nrpe migration story by being able to use the nrpe scripts more or less unchanged.
Problem statement
Icinga checks can contain arbitrary logic since the execution model is "exec the program and alert based on its exit status". Over time we have written, deployed and tested checks with complex logic to ensure services work as expected.
As we migrate away from Icinga and onto Prometheus/Alertmanager some of these checks can be rewritten to be based on service metrics instead. However for some we can't do it now (e.g. no service metrics yet, no engineering resources, etc) and thus we have to support an "icinga compatibility layer".
In practical terms this means for puppet users to be able to run an icinga check on their hosts and get an alertmanager alert back if something goes wrong (i.e. exit status != 0). To support the current use cases the alert should also include the first line of stdout/stderr.
Scope
Checks with significant and complex logic are in scope to be used by the compatibility layer. Simple checks or checks with modern replacements are not in scope. (e.g. check_procs nowadays is covered by systemd metrics and SystemdUnitFailed alert or check_http is covered by blackbox checks and their alerts).
Something else to consider is checks for which we use icinga's event_handler, i.e. raid checks where we use the event handler to gather more information/context to be included in the task description. The options below do not currently cater for said use case which we'll have to think about separately.
Implementation options
1. Dedicated exporter
With this option we have the following:
- https://github.com/canonical/nrpe_exporter deployed to the host, which talks to the local nrpe server
- puppet builds the exporter configuration, configuring how to run each script
- puppet configures prometheus to scrape said script
- puppet deploys a corresponding prometheus alert for metrics
Pros
- nrpe exporter is already written and tested
- the nrpe::monitor_service puppet code can stay in place as-is, since we would be essentially talking to nrpe-server instead of icinga
Cons
- nrpe-server keeps running on the hosts, which is literally remote command execution
- command output is not reported in metrics and thus in the alert
- we need to scrape (i.e. run the nrpe commands) at most every five minutes to have continuous metrics on prometheus
2. Using node-exporter textfile wrapper
With this solution we develop a wrapper to execute the given script and capture its exit status, and stdout/stderr. The metrics are written in Prometheus format on the filesystem for node-exporter to pick up. A systemd timer periodically executes the wrapper + script.
Since the stdout/stderr are arbitrary strings they will need to be included as label values, and space on label values is limited. For this reason the idea with this solution is to include the first line of stderr/stdout as label value, for example:
check_exit_status{name="check_foo"} 1
check_output{name="check_foo", stdout="first line of stdout", stderr="first line of stderr"} 1Pros
- nrpe is out of the picture
- scripts can be executed with any cadence (i.e. not max every 5 minutes)
- stdout/stderr can be included in alert text
Cons
- some (not a lot) code to be written and tested for said wrapper
- more changes at puppet level
- potentially high cardinality metrics if output changes often
See T384472: Candidate nrpe checks for compatibility layer icinga/prometheus/alertmanager for a list/audit of candidates checks/scripts
3. Modify the remaining check_ scripts to emit metrics to a push gateway
This option offers the same pros and cons as option 2 with the benefit of not needing a wrapper.
Pros
- nrpe is out of the picture
- scripts can be executed with any cadence (i.e. not max every 5 minutes)
- stdout/stderr can be included in alert text
- Doesn't need a wrapper.
Cons
- some (not a lot) code to be written and tested for said wrapper
- more changes at puppet level
- potentially high cardinality metrics if output changes often