We need availability monitoring of Cassandra that goes beyond monitoring the process. Though unlikely, it's possible for the process to be present, even though the node is unable to answer a query. If that were to happen now, we'd only see this via the errors piling up in RESTBase, or as a degradation of performance.
In short, a TCP port check of 9042 would be better, a custom check that ran a simple CQL query (for example: select host_id from system.local limit 1) would be better still.
| Status | Subtype | Assigned | Task | |
|---|---|---|---|---|
| · · · | ||||
| Resolved | fgiunchedi | T78514 Detailed cassandra monitoring: metrics and dashboards done, need to set up alerts | ||
| Resolved | fgiunchedi | T93886 Cassandra/CQL query interface monitoring | ||
| · · · |
Change 200242 had a related patch set uploaded (by Dzahn):
cassandra: add monitoring for CQL interface
added TCP port check to Icinga:
https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=cql
thanks @Dzahn ! I'm tentatively closing this as resolved, we can revisit the deeper cql query check if this isn't working well enough
Reopening to add a CQL-based check (since how we do Icinga service monitoring needs to be revisited in the wake of T95253: Finish conversion to multiple Cassandra instances per hardware node).
+1, nice work @Eevans. One minor nit: our icinga checks do not usually contain the pipe between the status text and the description in the output message.
LGTM overall! some points:
According to the docs, the | separates the status text from performance data. @fgiunchedi has said we don't make use of that (I wasn't sure), but didn't think it could hurt either way.
Good point; I'll take care of that.
- as @mobrovac pointed above "performance data" in icinga parlance is optional (IIRC we're not using it)
Do you think I should remove it?
- since there are dependencies to be included deployment will need some thought
Yeah. :(
How about this? https://github.com/eevans/icinga-checkcql/commit/4ff15cd6cd334f5822ffeaf347b9580a1abcddc0
$ ./check.js -H localhost CRITICAL | connect=NaN; execute=NaN; connect(): Error: All host(s) tried for query failed. First host tried, localhost:9042: Error: Connection timeout. See innerErrors.
Change 250439 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: multi-instance aware CQL checks
Change 250439 merged by Filippo Giunchedi:
cassandra: multi-instance aware CQL checks
myself and @Eevans have chatted about this a bit and decided that deepening the check from tcp-only to cql was out of scope for now. the tcp check became multi-instance aware in https://gerrit.wikimedia.org/r/250439 thus resolving