Page MenuHomePhabricator

Alert UNKNOWN for restbase cassandra graphite alerts
Closed, ResolvedPublic

Description

A number of check_graphite-based Icinga alerts are reporting UNKNOWN since RESTBase was converted to Prometheus. Now is a good time to reevaluate these alerts, and convert the those that are useful to use check_prometheus, instead.

  • RESTBase Cassandra highest tombstones scanned
  • RESTBase Cassandra highest dropped message rate
  • RESTBase Cassandra highest pending compactions
  • RESTBase Cassandra highest pending internal thread pool tasks
  • RESTBase Cassandra highest storage exceptions
  • RESTBase Cassandra highest total hints

Event Timeline

fgiunchedi triaged this task as Normal priority.Jan 17 2018, 10:43 AM
fgiunchedi created this task.

This is true for other restbase-cassandra graphite based alerts now:

RESTBase Cassandra highest dropped message rate
UNKNOWN 2018-01-31 09:49:18 6d 13h 58m 48s 3/3 UNKNOWN: No valid datapoints found

RESTBase Cassandra highest pending compactions
UNKNOWN 2018-01-31 09:48:48 6d 13h 32m 22s 3/3 UNKNOWN: No valid datapoints found

RESTBase Cassandra highest pending internal thread pool tasks
UNKNOWN 2018-01-31 09:49:01 6d 13h 48m 24s 3/3 UNKNOWN: No valid datapoints found

RESTBase Cassandra highest storage exceptions
UNKNOWN 2018-01-31 09:49:21 6d 13h 49m 44s 3/3 UNKNOWN: No valid datapoints found

RESTBase Cassandra highest total hints
UNKNOWN 2018-01-31 09:49:19 6d 13h 52m 32s 3/3 UNKNOWN: No valid datapoints found

fgiunchedi renamed this task from Alert UNKNOWN: RESTBase Cassandra highest tombstones scanned to Alert UNKNOWN for restbase cassandra graphite alerts.Jan 31 2018, 9:52 AM
Eevans claimed this task.Feb 2 2018, 5:17 PM
Eevans moved this task from Backlog to Next on the User-Eevans board.Feb 2 2018, 5:23 PM
Eevans updated the task description. (Show Details)Feb 16 2018, 9:20 PM
mobrovac changed the task status from Open to Stalled.Mar 28 2019, 3:44 PM

Setting as stalled since we are not actively working on this, but not closing it for visibility.

Change 525856 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Remove RESTBase graphite alerts.

https://gerrit.wikimedia.org/r/525856

Change 525856 merged by Effie Mouzeli:
[operations/puppet@production] Remove RESTBase graphite alerts.

https://gerrit.wikimedia.org/r/525856

Pchelolo closed this task as Resolved.Aug 12 2019, 4:41 PM