Page MenuHomePhabricator
Create Task
Maniphest T185089

Alert UNKNOWN for restbase cassandra graphite alerts
Closed, ResolvedPublic
Actions

Description

A number of check_graphite-based Icinga alerts are reporting UNKNOWN since RESTBase was converted to Prometheus. Now is a good time to reevaluate these alerts, and convert the those that are useful to use check_prometheus, instead.

  • RESTBase Cassandra highest tombstones scanned
  • RESTBase Cassandra highest dropped message rate
  • RESTBase Cassandra highest pending compactions
  • RESTBase Cassandra highest pending internal thread pool tasks
  • RESTBase Cassandra highest storage exceptions
  • RESTBase Cassandra highest total hints

Details

SubjectRepoBranchLines +/-
Remove RESTBase graphite alerts.operations/puppetproduction+7 -126
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi triaged this task as Medium priority.Jan 17 2018, 10:43 AM
fgiunchedi created this task.
Eevans added a project: User-Eevans.Jan 17 2018, 3:24 PM
fgiunchedi added a comment.Jan 31 2018, 9:52 AM

This is true for other restbase-cassandra graphite based alerts now:

RESTBase Cassandra highest dropped message rate
UNKNOWN 2018-01-31 09:49:18 6d 13h 58m 48s 3/3 UNKNOWN: No valid datapoints found

RESTBase Cassandra highest pending compactions
UNKNOWN 2018-01-31 09:48:48 6d 13h 32m 22s 3/3 UNKNOWN: No valid datapoints found

RESTBase Cassandra highest pending internal thread pool tasks
UNKNOWN 2018-01-31 09:49:01 6d 13h 48m 24s 3/3 UNKNOWN: No valid datapoints found

RESTBase Cassandra highest storage exceptions
UNKNOWN 2018-01-31 09:49:21 6d 13h 49m 44s 3/3 UNKNOWN: No valid datapoints found

RESTBase Cassandra highest total hints
UNKNOWN 2018-01-31 09:49:19 6d 13h 52m 32s 3/3 UNKNOWN: No valid datapoints found

fgiunchedi renamed this task from Alert UNKNOWN: RESTBase Cassandra highest tombstones scanned to Alert UNKNOWN for restbase cassandra graphite alerts.Jan 31 2018, 9:52 AM
Eevans claimed this task.Feb 2 2018, 5:17 PM
Eevans moved this task from Backlog to Next on the User-Eevans board.Feb 2 2018, 5:23 PM
Eevans updated the task description. (Show Details)Feb 16 2018, 9:20 PM
• mobrovac added a project: Platform Team Workboards (Doing).Dec 20 2018, 12:43 PM
• mobrovac changed the task status from Open to Stalled.Mar 28 2019, 3:44 PM
• mobrovac edited projects, added Services (later), Platform Team Legacy (Later); removed Platform Team Workboards (Doing), Services (doing).

Setting as stalled since we are not actively working on this, but not closing it for visibility.

WDoranWMF edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Team Legacy (Later), Services (later).Jul 18 2019, 8:08 PM
gerritbot added a comment.Jul 26 2019, 6:04 PM

Change 525856 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Remove RESTBase graphite alerts.

https://gerrit.wikimedia.org/r/525856

gerritbot added a project: Patch-For-Review.Jul 26 2019, 6:04 PM
• Pchelolo moved this task from Inbox to Waiting for Review on the Platform Team Workboards (Clinic Duty Team) board.Jul 26 2019, 6:05 PM
• Pchelolo mentioned this in T229137: Create grafana alerts for RESTBase.Jul 26 2019, 6:09 PM
• Pchelolo moved this task from Waiting for Review to Blocked Externally on the Platform Team Workboards (Clinic Duty Team) board.Jul 29 2019, 11:18 PM
• Pchelolo merged a task: T164660: Restbase graphite alerts often showing up as UNKNOWN in icinga.Jul 31 2019, 9:56 PM
• Pchelolo added subscribers: elukey, Eevans.
gerritbot added a comment.Aug 12 2019, 1:35 PM

Change 525856 merged by Effie Mouzeli:
[operations/puppet@production] Remove RESTBase graphite alerts.

https://gerrit.wikimedia.org/r/525856

Maintenance_bot removed a project: Patch-For-Review.Aug 12 2019, 2:10 PM
• Pchelolo closed this task as Resolved.Aug 12 2019, 4:41 PM
WDoranWMF moved this task from Blocked Externally to Done on the Platform Team Workboards (Clinic Duty Team) board.Aug 13 2019, 6:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL