We have (have had for some time) abnormally wide partitions in Cassandra. These are the source of a number of problems, not least of which are fatally large heap allocations that result in [[ https://phabricator.wikimedia.org/T140946 | OOMs ]] when read.
We should a) find those that currently exist and clean them up, and b) put in place the means to proactively identify them moving forward.
Perhaps the lowest hanging of fruit can be found by grepping the logs created by compaction. For example, here are a few that I found in the logs on rack 'b' in eqiad, that are > ~10G in size.
```
10258736013 local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data:de.wikipedia.org:Wikipedia\:Auskunft
10572125663 local_group_wikipedia_T_parsoid_html/data:zh.wikipedia.org:Wikipedia\:新条目推荐/候选
10933494277 local_group_wikipedia_T_parsoid_html/data:en.wikipedia.org:User\:Onel5969/CSD_log
11541401765 local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data:en.wikipedia.org:Wikipedia\:Administrators'_noticeboard/Incidents
12103727258 local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data:sv.wikipedia.org:Användare\:Lsjbot/Anomalier-PRIVAT
12580672532 local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data:ceb.wikipedia.org:Gumagamit\:Lsjbot/Anomalier-PRIVAT
12913411524 local_group_wikimedia_T_parsoid_html/data:commons.wikimedia.org:Commons\:Quality_images_candidates/candidate_list
12930627380 local_group_wikipedia_T_parsoid_html/data:sv.wikipedia.org:Användare\:Lsjbot/Anomalier-PRIVAT
13311461441 local_group_wikipedia_T_parsoid_html/data:en.wikipedia.org:User\:EranBot/Copyright/rc
14767803922 local_group_wikipedia_T_parsoid_html/data:ceb.wikipedia.org:Gumagamit\:Lsjbot/Anomalier-PRIVAT
15762884884 local_group_wikipedia_T_parsoid_html/data:de.wikipedia.org:Wikipedia\:Auskunft
15853965607 local_group_wikipedia_T_parsoid_html/data:nl.wikipedia.org:Gebruiker\:BartBotje/Kladblok
16437244256 local_group_wikipedia_T_parsoid_html/data:de.wikipedia.org:Wikipedia\:Auskunft
16853150229 local_group_wikipedia_T_parsoid_html/data:en.wikipedia.org:2016_in_sports
18224209523 local_group_wikimedia_T_parsoid_html/data:commons.wikimedia.org:User\:Oxyman/Buildings_in_London/2014_October_11-20
22398961158 local_group_wikipedia_T_parsoid_html/data:en.wikipedia.org:Wikipedia\:Administrators'_noticeboard/Incidents
```