Think about / discuss (e.g. with Foundations) re: “host down” semantics.
Can we move to “we can’t collect metrics from this host” as opposed to “ssh is down”?
Non-hosts (PDU, routers, etc) can stay on icmp checks for now.
| Migrated?(Y/N) | Title | Resource Type | Command | File | Profiles |
|---|---|---|---|---|---|
| N | ssh | Monitoring::Service | check_ssh | /srv/puppet_code/environments/production/modules/monitoring/manifests/init.pp:16 | profile::base::production, profile::monitoring |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | tappof | T395441 Port all Icinga checks to Prometheus/Alertmanager: preparation | |||
| Open | Goal | None | T395448 Discuss about "host down" semantics |
Change #1195693 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):
[operations/puppet@production] monitoring services: add migration task T395448 to instances
Change #1195693 merged by Tiziano Fogli:
[operations/puppet@production] monitoring services: add migration task T395448 to instances
I'm starting to see some hosts where the SSH alert is the only remaining Icinga-based check (in addition to the implied host ping check), so I'd love to find a way to get rid of it fleet-wide. sshd itself is quite reliable and I'm not sure if I've ever seen this alert fire for ssh-related reasons on any other scenario except the bastions getting DoSed in some way. So I'd like to propose either porting the Icinga alert as-is (for now) or replacing it with a Blackbox TCP probe on the bastion hosts, and removing it for the remaining ones. Thoughts?
Agreed re: 'sshd is quite reliable enough'.
Given how much we rely on host metrics, I am +1 on defining an "host down" as "prometheus has not been able to collect (node-exporter only?) metrics for this host" and ditch the ssh check altogether