While investigating whether it was safe to do a decommission (T119935: Upgrade restbase100[7-9] to match restbase100[1-6] hardware), a number of large (100GB+) sstables were found, mostly belonging to local_group_wikipedia_T_parsoid_html and local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ
eqiad
restbase1001.eqiad.wmnet: 127495153070 Jan 26 11:15 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data-f4d92a60c2cb11e4ab6181ba0e170b9f/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ-data-ka-502803-Data.db restbase1001.eqiad.wmnet: 388938516642 Jan 23 06:49 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1032092-Data.db restbase1001.eqiad.wmnet: 185769479012 Jan 27 17:10 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1038636-Data.db restbase1001.eqiad.wmnet: 111310087737 Jan 9 10:54 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1022273-Data.db restbase1002.eqiad.wmnet: 162380600851 Feb 1 08:57 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1050123-Data.db restbase1002.eqiad.wmnet: 409211605648 Feb 1 11:18 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1048802-Data.db restbase1002.eqiad.wmnet: 186547550850 Jan 9 19:05 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1026199-Data.db restbase1002.eqiad.wmnet: 151159969823 Jan 22 04:14 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data-f4d92a60c2cb11e4ab6181ba0e170b9f/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ-data-ka-469143-Data.db restbase1003.eqiad.wmnet: 334065628159 Feb 2 01:15 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-994255-Data.db restbase1003.eqiad.wmnet: 135789307038 Jan 9 10:18 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-971574-Data.db restbase1003.eqiad.wmnet: 124779613448 Feb 1 01:48 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data-f4d92a60c2cb11e4ab6181ba0e170b9f/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ-data-ka-508414-Data.db restbase1004.eqiad.wmnet: 210490091937 Jan 10 00:17 /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-12926-Data.db restbase1004.eqiad.wmnet: 320535565084 Feb 2 00:38 /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-36231-Data.db restbase1004.eqiad.wmnet: 177687145087 Jan 21 18:56 /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-23630-Data.db restbase1004.eqiad.wmnet: 111817799238 Jan 21 03:28 /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data-f4d92a60c2cb11e4ab6181ba0e170b9f/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ-data-ka-9851-Data.db restbase1005.eqiad.wmnet: 162924636480 Jan 28 09:46 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1069227-Data.db restbase1005.eqiad.wmnet: 182203048061 Jan 9 17:07 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1051364-Data.db restbase1005.eqiad.wmnet: 402386944808 Feb 2 16:48 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1074055-Data.db restbase1005.eqiad.wmnet: 136095741714 Jan 26 10:28 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data-f4d92a60c2cb11e4ab6181ba0e170b9f/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ-data-ka-570519-Data.db restbase1006.eqiad.wmnet: 172080229432 Jan 17 19:38 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-995725-Data.db restbase1006.eqiad.wmnet: 145445796927 Jan 27 23:40 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1005632-Data.db restbase1006.eqiad.wmnet: 354055854836 Jan 31 23:43 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1008838-Data.db restbase1006.eqiad.wmnet: 122708401314 Jan 31 23:28 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data-f4d92a60c2cb11e4ab6181ba0e170b9f/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ-data-ka-447000-Data.db restbase1007.eqiad.wmnet: 108110540027 Feb 6 12:26 /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-20091-Data.db restbase1007.eqiad.wmnet: 158879625023 Jan 31 05:08 /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-17301-Data.db restbase1008.eqiad.wmnet: 109558289101 Feb 7 05:28 /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-18706-Data.db restbase1009.eqiad.wmnet: 208762968302 Jan 21 04:52 /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-18634-Data.db
codfw
restbase2002.codfw.wmnet: 112537008879 Jan 20 10:27 /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-21051-Data.db restbase2003.codfw.wmnet: 218131112901 Jan 9 22:39 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-182511-Data.db restbase2003.codfw.wmnet: 229355616392 Feb 1 05:23 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-208453-Data.db restbase2003.codfw.wmnet: 210988335257 Jan 21 20:31 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-194773-Data.db restbase2004.codfw.wmnet: 184258575748 Jan 21 17:37 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-196682-Data.db restbase2004.codfw.wmnet: 256760687053 Feb 1 07:02 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-211174-Data.db restbase2004.codfw.wmnet: 231210786676 Jan 9 21:46 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-184135-Data.db restbase2004.codfw.wmnet: 111108207450 Jan 21 04:03 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ/data-f4d92a60c2cb11e4ab6181ba0e170b9f/local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ-data-ka-64034-Data.db restbase2005.codfw.wmnet: 226636152901 Jan 9 22:51 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-190601-Data.db restbase2005.codfw.wmnet: 248658084363 Feb 1 00:28 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-217328-Data.db restbase2005.codfw.wmnet: 209888122903 Jan 22 01:02 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-203336-Data.db restbase2006.codfw.wmnet: 244311982871 Jan 30 04:48 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-215379-Data.db restbase2006.codfw.wmnet: 121523042580 Jan 9 12:16 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-192351-Data.db restbase2006.codfw.wmnet: 252504850245 Jan 22 04:50 /var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-204320-Data.db
In particular, what prompted this investigation was that the ~400GB file on restbase1002 (/var/lib/cassandra/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-1048802-Data.db), resulted in a ~20% swing in disk space utilized during its compaction. The upper bound on these files is determined by a time interval that is currently 818 days, since these very large files have intervals of less than 1 year, they are on-track to become considerably larger. Given that we must maintain at least as much free space as the largest sstable, the current trajectory is intractable, even in light of upcoming smaller instance sizes.
Additionally, an evaluation of the files (as they looked on Feb. 2), would seem to indicate that compaction is failing to produce the date-based tiering expected.