Instead of having nrpe dependent alerts to go critical during an nrpe interruption (stopped nrpe daemon, etc.) we should consider using unknown status for this condition via the check_nrpe -u flag.
-u = Make socket timeouts return an UNKNOWN state instead of CRITICAL
This, in conjunction with an a service check for nrpe itself, could significantly reduce false critical alerts.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| icinga: add -u option to check_nrpe commands | operations/puppet | production | +13 -13 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T152967 Investigate usage of service dependencies in icinga | |||
| Resolved | herron | T172131 Investigate check_nrpe -u option to reduce critical alerts |
Change 374368 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] icinga: add -u option to check_nrpe commands
Change 374368 merged by Herron:
[operations/puppet@production] icinga: add -u option to check_nrpe commands
This is looking good so far. Going to keep an eye on it for the rest of the day before resolving