Change Details

In the process of deprecating/removing icinga we have a few scripts invoked via nrpe (i.e. locally on the host, as opposed on the icinga host). It would be convenient to at least evaluate if we can ease the icinga nrpe migration story by being able to use the nrpe scripts more or less unchanged. ## Problem statement Icinga checks can contain arbitrary logic since the execution model is "exec the program and alert based on its exit status". Over time we have written, deployed and tested checks with complex logic to ensure services work as expected. As we migrate away from Icinga and onto Prometheus/Alertmanager some of these checks can be rewritten to be based on service metrics instead. However for some we can't do it now (e.g. no service metrics yet, no engineering resources, etc) and thus we have to support an "icinga compatibility layer". In practical terms this means for puppet users to be able to run an icinga check on their hosts and get an alertmanager alert back if something goes wrong (i.e. exit status != 0). To support the current use cases the alert should also include the first line of stdout/stderr. ## Scope Checks with significant and complex logic are in scope to be used by the compatibility layer. Simple checks or checks with modern replacements are not in scope. (e.g. `check_procs` nowadays is covered by systemd metrics and SystemdUnitFailed alert or `check_http` is covered by blackbox checks and their alerts) ## Implementation options ###1. Dedicated exporter With this option we have the following: 1. https://github.com/canonical/nrpe_exporter deployed to the host, which talks to the local nrpe server 1. puppet builds the exporter configuration, configuring how to run each script 1. puppet configures prometheus to scrape said script 1. puppet deploys a corresponding prometheus alert for metrics ######Pros 1. nrpe exporter is already written and tested 1. the `nrpe::monitor_service` puppet code can stay in place as-is, since we would be essentially talking to nrpe-server instead of icinga ######Cons 1. nrpe-server keeps running on the hosts, which is literally remote command execution 1. command output is not reported in metrics and thus in the alert 1. we need to scrape (i.e. run the nrpe commands) at most every five minutes to have continuous metrics on prometheus #####2. Using node-exporter textfile wrapper With this solution we develop a wrapper to execute the given script and capture its exit status, and stdout/stderr. The metrics are written in Prometheus format on the filesystem for node-exporter to pick up. A systemd timer periodically executes the wrapper + script. ######Pros 1. nrpe is out of the picture 1. scripts can be executed with any cadence (i.e. not max every 5 minutes) 1. stdout/stderr can be included in alert text ######Cons 1. some (not a lot) code to be written and tested for said wrapper 1. more changes at puppet level 1. potentially high cardinality metrics if output changes often See {T384472} for a list/audit of candidates checks/scripts

In the process of deprecating/removing icinga we have a few scripts invoked via nrpe (i.e. locally on the host, as opposed on the icinga host). It would be convenient to at least evaluate if we can ease the icinga nrpe migration story by being able to use the nrpe scripts more or less unchanged. ## Problem statement Icinga checks can contain arbitrary logic since the execution model is "exec the program and alert based on its exit status". Over time we have written, deployed and tested checks with complex logic to ensure services work as expected. As we migrate away from Icinga and onto Prometheus/Alertmanager some of these checks can be rewritten to be based on service metrics instead. However for some we can't do it now (e.g. no service metrics yet, no engineering resources, etc) and thus we have to support an "icinga compatibility layer". In practical terms this means for puppet users to be able to run an icinga check on their hosts and get an alertmanager alert back if something goes wrong (i.e. exit status != 0). To support the current use cases the alert should also include the first line of stdout/stderr. ## Scope Checks with significant and complex logic are in scope to be used by the compatibility layer. Simple checks or checks with modern replacements are not in scope. (e.g. `check_procs` nowadays is covered by systemd metrics and SystemdUnitFailed alert or `check_http` is covered by blackbox checks and their alerts) ## Implementation options ### 1. Dedicated exporter With this option we have the following: 1. https://github.com/canonical/nrpe_exporter deployed to the host, which talks to the local nrpe server 1. puppet builds the exporter configuration, configuring how to run each script 1. puppet configures prometheus to scrape said script 1. puppet deploys a corresponding prometheus alert for metrics ######Pros 1. nrpe exporter is already written and tested 1. the `nrpe::monitor_service` puppet code can stay in place as-is, since we would be essentially talking to nrpe-server instead of icinga ######Cons 1. nrpe-server keeps running on the hosts, which is literally remote command execution 1. command output is not reported in metrics and thus in the alert 1. we need to scrape (i.e. run the nrpe commands) at most every five minutes to have continuous metrics on prometheus ### 2. Using node-exporter textfile wrapper With this solution we develop a wrapper to execute the given script and capture its exit status, and stdout/stderr. The metrics are written in Prometheus format on the filesystem for node-exporter to pick up. A systemd timer periodically executes the wrapper + script. ######Pros 1. nrpe is out of the picture 1. scripts can be executed with any cadence (i.e. not max every 5 minutes) 1. stdout/stderr can be included in alert text ######Cons 1. some (not a lot) code to be written and tested for said wrapper 1. more changes at puppet level 1. potentially high cardinality metrics if output changes often See {T384472} for a list/audit of candidates checks/scripts ### 3. Modify the remaining check_ scripts to emit metrics to a push gateway This option offers the same pros and cons as option 2 with the benefit of not needing a wrapper. ######Pros 1. nrpe is out of the picture 1. scripts can be executed with any cadence (i.e. not max every 5 minutes) 1. stdout/stderr can be included in alert text 1. Doesn't need a wrapper. ######Cons 1. some (not a lot) code to be written and tested for said wrapper 1. more changes at puppet level 1. potentially high cardinality metrics if output changes often

In the process of deprecating/removing icinga we have a few scripts invoked via nrpe (i.e. locally on the host, as opposed on the icinga host). It would be convenient to at least evaluate if we can ease the icinga nrpe migration story by being able to use the nrpe scripts more or less unchanged. ## Problem statement Icinga checks can contain arbitrary logic since the execution model is "exec the program and alert based on its exit status". Over time we have written, deployed and tested checks with complex logic to ensure services work as expected. As we migrate away from Icinga and onto Prometheus/Alertmanager some of these checks can be rewritten to be based on service metrics instead. However for some we can't do it now (e.g. no service metrics yet, no engineering resources, etc) and thus we have to support an "icinga compatibility layer". In practical terms this means for puppet users to be able to run an icinga check on their hosts and get an alertmanager alert back if something goes wrong (i.e. exit status != 0). To support the current use cases the alert should also include the first line of stdout/stderr. ## Scope Checks with significant and complex logic are in scope to be used by the compatibility layer. Simple checks or checks with modern replacements are not in scope. (e.g. `check_procs` nowadays is covered by systemd metrics and SystemdUnitFailed alert or `check_http` is covered by blackbox checks and their alerts) ## Implementation options ### 1. Dedicated exporter With this option we have the following: 1. https://github.com/canonical/nrpe_exporter deployed to the host, which talks to the local nrpe server 1. puppet builds the exporter configuration, configuring how to run each script 1. puppet configures prometheus to scrape said script 1. puppet deploys a corresponding prometheus alert for metrics ######Pros 1. nrpe exporter is already written and tested 1. the `nrpe::monitor_service` puppet code can stay in place as-is, since we would be essentially talking to nrpe-server instead of icinga ######Cons 1. nrpe-server keeps running on the hosts, which is literally remote command execution 1. command output is not reported in metrics and thus in the alert 1. we need to scrape (i.e. run the nrpe commands) at most every five minutes to have continuous metrics on prometheus #####2### 2. Using node-exporter textfile wrapper With this solution we develop a wrapper to execute the given script and capture its exit status, and stdout/stderr. The metrics are written in Prometheus format on the filesystem for node-exporter to pick up. A systemd timer periodically executes the wrapper + script. ######Pros 1. nrpe is out of the picture 1. scripts can be executed with any cadence (i.e. not max every 5 minutes) 1. stdout/stderr can be included in alert text ######Cons 1. some (not a lot) code to be written and tested for said wrapper 1. more changes at puppet level 1. potentially high cardinality metrics if output changes often See {T384472} for a list/audit of candidates checks/scripts ### 3. Modify the remaining check_ scripts to emit metrics to a push gateway This option offers the same pros and cons as option 2 with the benefit of not needing a wrapper. ######Pros 1. nrpe is out of the picture 1. scripts can be executed with any cadence (i.e. not max every 5 minutes) 1. stdout/stderr can be included in alert text 1. Doesn't need a wrapper. ######Cons 1. some (not a lot) code to be written and tested for said wrapper 1. more changes at puppet level 1. potentially high cardinality metrics if output changes often