Page MenuHomePhabricator

Address abnormally wide partitions
Closed, DuplicatePublic

Description

We have (have had for some time) abnormally wide partitions in Cassandra. These are the source of a number of problems, not least of which are fatally large heap allocations that result in OOMs when read.

We should a) find those that currently exist and clean them up, and b) put in place the means to proactively identify them moving forward.

First pass

CountDescription
18Partitions larger than 10G in size
30> 5G and <= 10G in size
653> 1G and <= 5G in size

Working files



(raw log entries)



Event Timeline

Script to delete the partitions >= 10G.

(applied w/ cqlsh -f delete_big.cql)

Script to delete the partitions >= 5G and < 10G.

(applied w/ cqlsh -f delete_5-to-10.cql)

Eevans triaged this task as Medium priority.Aug 23 2016, 9:31 PM