Collect droppable tombstone ratio metrics
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Eevans
	Aug 12 2016, 3:38 PM

Description

The droppable tombstone ratio is shaping up to be an important metric for us. We should collect, persist, and configure dashboards for it.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Eevans	T140008 High RESTBase storage utilization
		Resolved		Eevans	T142844 Collect droppable tombstone ratio metrics

Event Timeline

Eevans created this task.Aug 12 2016, 3:38 PM

Eevans moved this task from Backlog to In-Progress on the Cassandra board.Aug 12 2016, 9:25 PM

A couple of problems with this:

a) The metric is not included in o.a.cassandra.metrics, so as things stand, we cannot get at it with cassandra-metrics-collector, and b) each query is roughly equivalent to invoking sstablemetadata against each file, so reading it every collection interval would be too expensive (it doesn't change often enough to warrant that anyway). So for the time being, I will collect some ad hoc results using something like the following

P3825 Masterwork From Distant Lands

1	#!/bin/bash
2
3	# https://phabricator.wikimedia.org/P3825
4
5	set -ex
6
7
8	export PATH="$PATH:~eevans/c-commands"
9
10	connection_string()
11	{
12	printf "localhost:%d" $(uyaml /etc/cassandra-instances.d/`hostname`-"$1".yaml /jmx_port)
13	}
14
15	bean()
16	{
17	printf "org.apache.cassandra.db:type=ColumnFamilies,keyspace=%s,columnfamily=data" "$1"
18	}
19
20	keyspaces()
21	{
22	c-cqlsh a -e "describe keyspaces" \| sed "s/\"//g" \| grep -E "^local_group_"
23	}
24
25	finish()
26	{
27	rm -f $OUTPUT
28	}
29
30	trap finish EXIT
31
32	OUTPUT=${1:-`mktemp`}
33
34	for i in `keyspaces`; do
35	for j in `c-ls`; do
36	ratio=$(sjk mx -s `connection_string $j` -b `bean "$i"` -mg -f DroppableTombstoneRatio \|tail -n 1)
37	printf "%s-%s,%s,data,%.3f\n" `hostname` "$j" "$i" $ratio >> "$OUTPUT"
38	done
39	done
40
41	install -d ~/reports
42	mv "$OUTPUT" ~/reports/droppable_tombstones_`date -Idate`.csv
43

In T142844#2549411, @Eevans wrote:

A couple of problems with this:

a) The metric is not included in o.a.cassandra.metrics, so as things stand, we can get at it with cassandra-metrics-collector, and b) each query is roughly equivalent to invoking sstablemetadata against each file, so reading it every collection interval would be too expensive (it doesn't change often enough to warrant that anyway). So for the time being, I will collect some ad hoc results using something like the following

[ ... ]

First report attached here:

droppable_tombstones_2016-08-12_all.csv514 KBDownload

Eevans moved this task from In-Progress to Next on the Cassandra board.Sep 20 2016, 9:01 PM

I think the approach mentioned here could work for this as well. Collect data weekly and generate a summary to email to services@.

Eevans moved this task from Next to Backlog on the Cassandra board.Nov 29 2016, 9:29 PM

• GWicke moved this task from Backlog to later on the Services board.Jul 11 2017, 10:57 PM

• GWicke edited projects, added Services (later); removed Services.

• mobrovac added a project: Platform Team Legacy (Later).Dec 20 2018, 12:07 PM

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

Eevans lowered the priority of this task from Medium to Low.Jun 7 2021, 7:53 PM

I think our latest dashboards cover this scenario; Closing

	F4358011: droppable_tombstones_2016-08-12_all.csv
	Aug 12 2016, 9:48 PM

Collect droppable tombstone ratio metricsClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Collect droppable tombstone ratio metrics
Closed, ResolvedPublic
Actions

Related Objects
Search...