Page MenuHomePhabricator

Cassandra Grafana dashboards seem to disagree with actual utilization
Closed, ResolvedPublic

Description

Current storage utilization via Grafana would suggest ~30T of utilization out of ~76.5 total space, or ~39%. Actual disk utilization though seems to range from 60-78%.

eevans@hermes:$ cdsh -d eqiad -- "df -h |grep /srv/sd"
restbase1016.eqiad.wmnet: /dev/sdc4       1.4T  872G  548G  62% /srv/sdc4
restbase1016.eqiad.wmnet: /dev/sda4       1.4T  827G  594G  59% /srv/sda4
restbase1016.eqiad.wmnet: /dev/sdd4       1.4T  901G  519G  64% /srv/sdd4
restbase1016.eqiad.wmnet: /dev/sdb4       1.4T  850G  571G  60% /srv/sdb4
restbase1019.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  634G  63% /srv/sdb4
restbase1019.eqiad.wmnet: /dev/sda4       1.7T  1.1T  565G  67% /srv/sda4
restbase1019.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  553G  68% /srv/sdc4
restbase1020.eqiad.wmnet: /dev/sda4       1.7T  1.2T  468G  73% /srv/sda4
restbase1020.eqiad.wmnet: /dev/sdb4       1.7T  1.2T  485G  72% /srv/sdb4
restbase1020.eqiad.wmnet: /dev/sdc4       1.7T  1.3T  388G  78% /srv/sdc4
restbase1021.eqiad.wmnet: /dev/sda4       1.7T  1.1T  638G  63% /srv/sda4
restbase1021.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  578G  66% /srv/sdb4
restbase1021.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  516G  70% /srv/sdc4
restbase1017.eqiad.wmnet: /dev/sda4       1.4T  824G  596G  59% /srv/sda4
restbase1017.eqiad.wmnet: /dev/sdb4       1.4T  808G  612G  57% /srv/sdb4
restbase1017.eqiad.wmnet: /dev/sdc4       1.4T  830G  590G  59% /srv/sdc4
restbase1017.eqiad.wmnet: /dev/sdd4       1.4T  822G  598G  58% /srv/sdd4
restbase1022.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  527G  69% /srv/sdc4
restbase1022.eqiad.wmnet: /dev/sda4       1.7T  1.2T  510G  70% /srv/sda4
restbase1022.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  567G  67% /srv/sdb4
restbase1023.eqiad.wmnet: /dev/sdb4       1.7T  1.2T  539G  69% /srv/sdb4
restbase1023.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  546G  68% /srv/sdc4
restbase1023.eqiad.wmnet: /dev/sda4       1.7T  1.2T  482G  72% /srv/sda4
restbase1024.eqiad.wmnet: /dev/sda4       1.7T  1.1T  614G  64% /srv/sda4
restbase1024.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  615G  64% /srv/sdb4
restbase1024.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  540G  69% /srv/sdc4
restbase1018.eqiad.wmnet: /dev/sda4       1.4T  761G  659G  54% /srv/sda4
restbase1018.eqiad.wmnet: /dev/sdb4       1.4T  811G  610G  58% /srv/sdb4
restbase1018.eqiad.wmnet: /dev/sdc4       1.4T  773G  647G  55% /srv/sdc4
restbase1018.eqiad.wmnet: /dev/sdd4       1.4T  771G  649G  55% /srv/sdd4
restbase1025.eqiad.wmnet: /dev/sdc4       1.7T  1.1T  573G  67% /srv/sdc4
restbase1025.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  573G  67% /srv/sdb4
restbase1025.eqiad.wmnet: /dev/sda4       1.7T  1.1T  572G  67% /srv/sda4
restbase1026.eqiad.wmnet: /dev/sda4       1.7T  1.2T  510G  70% /srv/sda4
restbase1026.eqiad.wmnet: /dev/sdb4       1.7T  1.2T  494G  71% /srv/sdb4
restbase1026.eqiad.wmnet: /dev/sdc4       1.7T  1.3T  450G  74% /srv/sdc4
restbase1027.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  579G  66% /srv/sdb4
restbase1027.eqiad.wmnet: /dev/sda4       1.7T  1.1T  587G  66% /srv/sda4
restbase1027.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  555G  68% /srv/sdc4
eevans@hermes:$

As of 2020-09-29:

HostsStorage/host (nominal)TotalUsed
eqiad125T59.4T44.5 (75%)
codfw155T74.3T43.1 (58%)
NOTE: Grafana currently reports 33.22T and 34.75T for eqiad and codfw respectively.

See: T256863: restbase2009 down

Event Timeline

Eevans updated the task description. (Show Details)

It seems as though we do in fact have some snapshot data:

eevans@restbase1016:/srv$ sudo du -shc `sudo find sd[a-d]4 -name "dropped-*"`
16G	sda4/cassandra-b/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707062-data
8.0K	sda4/cassandra-b/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-1428763065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686781860-data
16G	sda4/cassandra-b/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672808-data
8.0K	sda4/cassandra-b/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-5e6971d065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686693491-meta
1.3G	sda4/cassandra-b/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808729-data
8.0K	sda4/cassandra-b/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-27a2b33065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686723398-meta
2.4G	sda4/cassandra-b/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841828-data
8.0K	sda4/cassandra-b/data/wikipedia_T_parsoid/meta-a8923c7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686768930-meta
47G	sda4/cassandra-b/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
7.1G	sda4/cassandra-b/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
8.0K	sda4/cassandra-b/data/others_T_parsoid/meta-f5f179f065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686738529-meta
8.0K	sda4/cassandra-a/data/others_T_references/meta-0d08bd40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172828813-meta
17G	sda4/cassandra-a/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707061-data
8.0K	sda4/cassandra-a/data/enwiki_T_parsoid/meta-343287f065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686709061-meta
8.0K	sda4/cassandra-a/data/commons_T_parsoid/meta-783bbef065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686677971-meta
17G	sda4/cassandra-a/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672806-data
12K	sda4/cassandra-a/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-5e6971d065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686693498-meta
8.0K	sda4/cassandra-a/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-c8e852b065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686691871-data
1.2G	sda4/cassandra-a/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808901-data
8.0K	sda4/cassandra-a/data/enwiki_T_references/meta-e432f160ae3e11e9bd29cd91111de967/snapshots/dropped-1585172813834-meta
2.6G	sda4/cassandra-a/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841829-data
12K	sda4/cassandra-a/data/wikipedia_T_references/meta-376b91c0ae3f11e9bd29cd91111de967/snapshots/dropped-1585172843706-meta
8.0K	sda4/cassandra-a/data/wikipedia_T_parsoid/meta-a8923c7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686768709-meta
45G	sda4/cassandra-a/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
7.1G	sda4/cassandra-a/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
8.0K	sda4/cassandra-c/data/others_T_references/meta-0d08bd40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172828964-meta
40K	sda4/cassandra-c/data/others_T_references/data-1023dff0ae3f11e9bd29cd91111de967/snapshots/dropped-1585172827410-data
11G	sda4/cassandra-c/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707048-data
8.0K	sda4/cassandra-c/data/enwiki_T_parsoid/meta-343287f065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686708954-meta
8.0K	sda4/cassandra-c/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-1428763065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686781860-data
8.0K	sda4/cassandra-c/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-082eaa7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686783413-meta
18G	sda4/cassandra-c/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672758-data
8.0K	sda4/cassandra-c/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-c8e852b065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686691871-data
1.2G	sda4/cassandra-c/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808714-data
8.0K	sda4/cassandra-c/data/enwiki_T_references/meta-e432f160ae3e11e9bd29cd91111de967/snapshots/dropped-1585172813561-meta
8.0K	sda4/cassandra-c/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-98246d6065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686721875-data
2.8G	sda4/cassandra-c/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841827-data
8.0K	sda4/cassandra-c/data/wikipedia_T_parsoid/meta-a8923c7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686768656-meta
43G	sda4/cassandra-c/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
4.7G	sda4/cassandra-c/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
8.0K	sda4/cassandra-c/data/others_T_parsoid/meta-f5f179f065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686738406-meta
40K	sdb4/cassandra-c/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-5e6971d065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686693366-meta
40K	sdb4/cassandra-c/data/wikipedia_T_references/meta-376b91c0ae3f11e9bd29cd91111de967/snapshots/dropped-1585172843615-meta
2.9G	sdb4/cassandra-c/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841827-data
44K	sdb4/cassandra-c/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-e4fc59d065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686753568-meta
40K	sdb4/cassandra-c/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-27a2b33065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686723316-meta
8.0K	sdb4/cassandra-c/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-98246d6065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686721875-data
40K	sdb4/cassandra-c/data/enwiki_T_parsoid/meta-343287f065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686708954-meta
21G	sdb4/cassandra-c/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707048-data
44K	sdb4/cassandra-c/data/others_T_references/meta-0d08bd40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172828964-meta
44K	sdb4/cassandra-c/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-082eaa7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686783413-meta
8.0K	sdb4/cassandra-c/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-1428763065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686781860-data
40K	sdb4/cassandra-c/data/others_T_parsoid/meta-f5f179f065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686738406-meta
7.9G	sdb4/cassandra-c/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
48K	sdb4/cassandra-c/data/commons_T_parsoid/meta-783bbef065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686677639-meta
18G	sdb4/cassandra-c/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672758-data
709M	sdb4/cassandra-c/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808714-data
40K	sdb4/cassandra-c/data/enwiki_T_references/meta-e432f160ae3e11e9bd29cd91111de967/snapshots/dropped-1585172813561-meta
40K	sdb4/cassandra-c/data/wikipedia_T_parsoid/meta-a8923c7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686768656-meta
48G	sdb4/cassandra-c/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
8.0K	sdb4/cassandra-a/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-c8e852b065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686691871-data
2.5G	sdb4/cassandra-a/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841829-data
12K	sdb4/cassandra-a/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-e4fc59d065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686753525-meta
8.0K	sdb4/cassandra-a/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-ea1d5ea065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686751881-data
8.0K	sdb4/cassandra-a/data/enwiki_T_parsoid/meta-343287f065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686709061-meta
18G	sdb4/cassandra-a/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707061-data
12K	sdb4/cassandra-a/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-1428763065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686781860-data
8.0K	sdb4/cassandra-a/data/others_T_parsoid/meta-f5f179f065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686738547-meta
7.4G	sdb4/cassandra-a/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
16G	sdb4/cassandra-a/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672806-data
1.3G	sdb4/cassandra-a/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808901-data
51G	sdb4/cassandra-a/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
8.0K	sdb4/cassandra-b/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-c8e852b065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686691880-data
12K	sdb4/cassandra-b/data/wikipedia_T_references/meta-376b91c0ae3f11e9bd29cd91111de967/snapshots/dropped-1585172843695-meta
2.7G	sdb4/cassandra-b/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841828-data
8.0K	sdb4/cassandra-b/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-e4fc59d065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686753501-meta
8.0K	sdb4/cassandra-b/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-ea1d5ea065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686751888-data
12K	sdb4/cassandra-b/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-98246d6065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686721875-data
8.0K	sdb4/cassandra-b/data/enwiki_T_parsoid/meta-343287f065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686709061-meta
12G	sdb4/cassandra-b/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707062-data
8.0K	sdb4/cassandra-b/data/others_T_references/data-1023dff0ae3f11e9bd29cd91111de967/snapshots/dropped-1585172827365-data
8.0K	sdb4/cassandra-b/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-082eaa7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686783376-meta
6.6G	sdb4/cassandra-b/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
8.0K	sdb4/cassandra-b/data/commons_T_parsoid/meta-783bbef065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686677690-meta
19G	sdb4/cassandra-b/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672808-data
1.2G	sdb4/cassandra-b/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808729-data
8.0K	sdb4/cassandra-b/data/enwiki_T_references/meta-e432f160ae3e11e9bd29cd91111de967/snapshots/dropped-1585172813375-meta
43G	sdb4/cassandra-b/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
8.0K	sdc4/cassandra-c/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-e4fc59d065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686753568-meta
8.0K	sdc4/cassandra-c/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-ea1d5ea065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686751881-data
2.8G	sdc4/cassandra-c/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841827-data
18G	sdc4/cassandra-c/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672758-data
8.0K	sdc4/cassandra-c/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-c8e852b065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686691871-data
8.0K	sdc4/cassandra-c/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-5e6971d065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686693366-meta
5.8G	sdc4/cassandra-c/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
8.0K	sdc4/cassandra-c/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-27a2b33065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686723316-meta
56G	sdc4/cassandra-c/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
8.0K	sdc4/cassandra-c/data/wikipedia_T_parsoid/meta-a8923c7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686768656-meta
1.2G	sdc4/cassandra-c/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808714-data
8.0K	sdc4/cassandra-c/data/enwiki_T_parsoid/meta-343287f065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686708954-meta
20G	sdc4/cassandra-c/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707048-data
2.6G	sdc4/cassandra-a/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841829-data
8.0K	sdc4/cassandra-a/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-082eaa7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686783314-meta
14G	sdc4/cassandra-a/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672806-data
6.9G	sdc4/cassandra-a/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
8.0K	sdc4/cassandra-a/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-98246d6065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686721875-data
8.0K	sdc4/cassandra-a/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-27a2b33065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686723289-meta
47G	sdc4/cassandra-a/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
8.0K	sdc4/cassandra-a/data/others_T_references/meta-0d08bd40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172828813-meta
1.3G	sdc4/cassandra-a/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808901-data
15G	sdc4/cassandra-a/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707061-data
8.0K	sdc4/cassandra-b/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-e4fc59d065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686753501-meta
8.0K	sdc4/cassandra-b/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-ea1d5ea065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686751888-data
2.5G	sdc4/cassandra-b/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841828-data
8.0K	sdc4/cassandra-b/data/commons_T_parsoid/meta-783bbef065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686677690-meta
19G	sdc4/cassandra-b/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672808-data
8.0K	sdc4/cassandra-b/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-c8e852b065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686691880-data
5.8G	sdc4/cassandra-b/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
8.0K	sdc4/cassandra-b/data/others_T_parsoid/meta-f5f179f065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686738529-meta
8.0K	sdc4/cassandra-b/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-27a2b33065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686723398-meta
49G	sdc4/cassandra-b/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
8.0K	sdc4/cassandra-b/data/wikipedia_T_parsoid/meta-a8923c7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686768930-meta
12K	sdc4/cassandra-b/data/others_T_references/meta-0d08bd40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172828946-meta
8.0K	sdc4/cassandra-b/data/others_T_references/data-1023dff0ae3f11e9bd29cd91111de967/snapshots/dropped-1585172827365-data
1.2G	sdc4/cassandra-b/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808729-data
8.0K	sdc4/cassandra-b/data/enwiki_T_parsoid/meta-343287f065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686709061-meta
14G	sdc4/cassandra-b/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707062-data
8.0K	sdd4/cassandra-b/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-1428763065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686781860-data
8.0K	sdd4/cassandra-b/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-082eaa7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686783376-meta
4.9G	sdd4/cassandra-b/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
18G	sdd4/cassandra-b/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672808-data
3.1G	sdd4/cassandra-b/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841828-data
50G	sdd4/cassandra-b/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
8.0K	sdd4/cassandra-b/data/enwiki_T_references/meta-e432f160ae3e11e9bd29cd91111de967/snapshots/dropped-1585172813375-meta
914M	sdd4/cassandra-b/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808729-data
14G	sdd4/cassandra-b/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707062-data
8.0K	sdd4/cassandra-b/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-5e6971d065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686693491-meta
8.0K	sdd4/cassandra-a/data/wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-082eaa7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686783314-meta
12K	sdd4/cassandra-a/data/others_T_references/data-1023dff0ae3f11e9bd29cd91111de967/snapshots/dropped-1585172827365-data
7.2G	sdd4/cassandra-a/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
8.0K	sdd4/cassandra-a/data/others_T_parsoid/meta-f5f179f065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686738547-meta
16G	sdd4/cassandra-a/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672806-data
8.0K	sdd4/cassandra-a/data/commons_T_parsoid/meta-783bbef065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686677971-meta
8.0K	sdd4/cassandra-a/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-27a2b33065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686723289-meta
8.0K	sdd4/cassandra-a/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-98246d6065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686721875-data
8.0K	sdd4/cassandra-a/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-ea1d5ea065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686751881-data
2.6G	sdd4/cassandra-a/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841829-data
47G	sdd4/cassandra-a/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
8.0K	sdd4/cassandra-a/data/wikipedia_T_parsoid/meta-a8923c7065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686768709-meta
8.0K	sdd4/cassandra-a/data/enwiki_T_references/meta-e432f160ae3e11e9bd29cd91111de967/snapshots/dropped-1585172813834-meta
1.3G	sdd4/cassandra-a/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808901-data
14G	sdd4/cassandra-a/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707061-data
12K	sdd4/cassandra-c/data/others_T_references/data-1023dff0ae3f11e9bd29cd91111de967/snapshots/dropped-1585172827410-data
7.9G	sdd4/cassandra-c/data/others_T_parsoid/data-fd2a3b3065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686736535-data
8.0K	sdd4/cassandra-c/data/others_T_parsoid/meta-f5f179f065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686738406-meta
18G	sdd4/cassandra-c/data/commons_T_parsoid/data-9d5b60a065dd11e9bc6c8958654eb0a6/snapshots/dropped-1578686672758-data
8.0K	sdd4/cassandra-c/data/enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-27a2b33065de11e9bc6c8958654eb0a6/snapshots/dropped-1578686723316-meta
8.0K	sdd4/cassandra-c/data/others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/data-ea1d5ea065da11e9bc6c8958654eb0a6/snapshots/dropped-1578686751881-data
12K	sdd4/cassandra-c/data/wikipedia_T_references/meta-376b91c0ae3f11e9bd29cd91111de967/snapshots/dropped-1585172843615-meta
2.9G	sdd4/cassandra-c/data/wikipedia_T_references/data-3aa86d40ae3f11e9bd29cd91111de967/snapshots/dropped-1585172841827-data
43G	sdd4/cassandra-c/data/wikipedia_T_parsoid/data-db05bba065db11e9bc6c8958654eb0a6/snapshots/dropped-1578686766856-data
8.0K	sdd4/cassandra-c/data/enwiki_T_references/meta-e432f160ae3e11e9bd29cd91111de967/snapshots/dropped-1585172813561-meta
1.2G	sdd4/cassandra-c/data/enwiki_T_references/data-e7a22780ae3e11e9bd29cd91111de967/snapshots/dropped-1585172808714-data
9.6G	sdd4/cassandra-c/data/enwiki_T_parsoid/data-ae7fe34065df11e9bc6c8958654eb0a6/snapshots/dropped-1578686707048-data
8.0K	sdd4/cassandra-c/data/commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU/meta-5e6971d065dc11e9bc6c8958654eb0a6/snapshots/dropped-1578686693366-meta
1.1T	total
eevans@restbase1016:/srv$

Despite that not showing up in the listsnapshots output:

eevans@restbase1016:/srv$ c-foreach-nt listsnapshots
a: Snapshot Details: 
a: There are no snapshots
b: Snapshot Details: 
b: There are no snapshots
c: Snapshot Details: 
c: There are no snapshots
eevans@restbase1016:/srv$

The following list of keyspaces seem to correspond with these ghost snapshots.

Allegedly obsolete keyspaces
commons_T_parsoid
commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
enwiki_T_parsoid
enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
enwiki_T_references
others_T_parsoid
others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
others_T_references
wikipedia_T_parsoid
wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
wikipedia_T_references

@Pchelolo can you check me on this before I go around rm -r-ing stuff?

The following list of keyspaces seem to correspond with these ghost snapshots.

Allegedly obsolete keyspaces
commons_T_parsoid
commons_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
enwiki_T_parsoid
enwiki_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
enwiki_T_references
others_T_parsoid
others_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
others_T_references
wikipedia_T_parsoid
wikipedia_T_parsoidd3o5Dn1wcj_Xve2tXe4_rtmeWSU
wikipedia_T_references

@Pchelolo can you check me on this before I go around rm -r-ing stuff?

@Pchelolo ping!

Mentioned in SAL (#wikimedia-operations) [2021-06-03T17:16:06Z] <urandom> remove dropped Cassandra keyspace snapshots -- T258414

In addition to removing the (orphaned) Cassandra keyspace snapshots, I also ran a round of cleanups, and the by-host utilization now looks like the following (scroll past the table for an explanation of current utilization):

hostfssizeutilization
restbase1016.eqiad.wmnet/dev/sda41.4T45%
restbase1016.eqiad.wmnet/dev/sdb41.4T48%
restbase1016.eqiad.wmnet/dev/sdc41.4T47%
restbase1016.eqiad.wmnet/dev/sdd41.4T50%
restbase1019.eqiad.wmnet/dev/sda41.7T53%
restbase1019.eqiad.wmnet/dev/sdb41.7T47%
restbase1019.eqiad.wmnet/dev/sdc41.7T53%
restbase1020.eqiad.wmnet/dev/sda41.7T55%
restbase1020.eqiad.wmnet/dev/sdb41.7T54%
restbase1020.eqiad.wmnet/dev/sdc41.7T52%
restbase1021.eqiad.wmnet/dev/sda41.7T49%
restbase1021.eqiad.wmnet/dev/sdb41.7T53%
restbase1021.eqiad.wmnet/dev/sdc41.7T45%
restbase1028.eqiad.wmnet/dev/sda41.7T46%
restbase1028.eqiad.wmnet/dev/sdb41.7T44%
restbase1028.eqiad.wmnet/dev/sdc41.7T46%
restbase1017.eqiad.wmnet/dev/sda41.4T46%
restbase1017.eqiad.wmnet/dev/sdb41.4T44%
restbase1017.eqiad.wmnet/dev/sdc41.4T46%
restbase1017.eqiad.wmnet/dev/sdd41.4T46%
restbase1022.eqiad.wmnet/dev/sda41.7T57%
restbase1022.eqiad.wmnet/dev/sdb41.7T49%
restbase1022.eqiad.wmnet/dev/sdc41.7T53%
restbase1023.eqiad.wmnet/dev/sda41.7T52%
restbase1023.eqiad.wmnet/dev/sdb41.7T41%
restbase1023.eqiad.wmnet/dev/sdc41.7T47%
restbase1024.eqiad.wmnet/dev/sda41.7T49%
restbase1024.eqiad.wmnet/dev/sdb41.7T47%
restbase1024.eqiad.wmnet/dev/sdc41.7T51%
restbase1029.eqiad.wmnet/dev/sda41.7T53%
restbase1029.eqiad.wmnet/dev/sdb41.7T51%
restbase1029.eqiad.wmnet/dev/sdc41.7T50%
restbase1018.eqiad.wmnet/dev/sda41.4T46%
restbase1018.eqiad.wmnet/dev/sdb41.4T40%
restbase1018.eqiad.wmnet/dev/sdc41.4T40%
restbase1018.eqiad.wmnet/dev/sdd41.4T46%
restbase1025.eqiad.wmnet/dev/sda41.7T52%
restbase1025.eqiad.wmnet/dev/sdb41.7T49%
restbase1025.eqiad.wmnet/dev/sdc41.7T49%
restbase1026.eqiad.wmnet/dev/sda41.7T49%
restbase1026.eqiad.wmnet/dev/sdb41.7T52%
restbase1026.eqiad.wmnet/dev/sdc41.7T57%
restbase1027.eqiad.wmnet/dev/sda41.7T48%
restbase1027.eqiad.wmnet/dev/sdb41.7T48%
restbase1027.eqiad.wmnet/dev/sdc41.7T50%
restbase1030.eqiad.wmnet/dev/sda41.7T51%
restbase1030.eqiad.wmnet/dev/sdb41.7T52%
restbase1030.eqiad.wmnet/dev/sdc41.7T52%
restbase2010.codfw.wmnet/dev/sda41.4T43%
restbase2010.codfw.wmnet/dev/sdb41.4T45%
restbase2010.codfw.wmnet/dev/sdc41.4T44%
restbase2010.codfw.wmnet/dev/sdd41.4T47%
restbase2013.codfw.wmnet/dev/sda41.7T51%
restbase2013.codfw.wmnet/dev/sdb41.7T48%
restbase2013.codfw.wmnet/dev/sdc41.7T50%
restbase2014.codfw.wmnet/dev/sda41.7T53%
restbase2014.codfw.wmnet/dev/sdb41.7T51%
restbase2014.codfw.wmnet/dev/sdc41.7T50%
restbase2019.codfw.wmnet/dev/sda41.7T47%
restbase2019.codfw.wmnet/dev/sdb41.7T43%
restbase2019.codfw.wmnet/dev/sdc41.7T53%
restbase2021.codfw.wmnet/dev/sda41.7T51%
restbase2021.codfw.wmnet/dev/sdb41.7T47%
restbase2021.codfw.wmnet/dev/sdc41.7T48%
restbase2011.codfw.wmnet/dev/sda41.4T45%
restbase2011.codfw.wmnet/dev/sdb41.4T43%
restbase2011.codfw.wmnet/dev/sdc41.4T47%
restbase2011.codfw.wmnet/dev/sdd41.4T47%
restbase2015.codfw.wmnet/dev/sda41.7T52%
restbase2015.codfw.wmnet/dev/sdb41.7T48%
restbase2015.codfw.wmnet/dev/sdc41.7T48%
restbase2016.codfw.wmnet/dev/sda41.7T51%
restbase2016.codfw.wmnet/dev/sdb41.7T45%
restbase2016.codfw.wmnet/dev/sdc41.7T45%
restbase2020.codfw.wmnet/dev/sda41.7T46%
restbase2020.codfw.wmnet/dev/sdb41.7T43%
restbase2020.codfw.wmnet/dev/sdc41.7T49%
restbase2022.codfw.wmnet/dev/sda41.7T49%
restbase2022.codfw.wmnet/dev/sdb41.7T44%
restbase2022.codfw.wmnet/dev/sdc41.7T46%
restbase2009.codfw.wmnet/dev/sda41.4T37%
restbase2009.codfw.wmnet/dev/sdb41.4T38%
restbase2009.codfw.wmnet/dev/sdc41.4T37%
restbase2009.codfw.wmnet/dev/sdd41.4T39%
restbase2012.codfw.wmnet/dev/sda41.4T44%
restbase2012.codfw.wmnet/dev/sdb41.4T43%
restbase2012.codfw.wmnet/dev/sdc41.4T43%
restbase2012.codfw.wmnet/dev/sdd41.4T49%
restbase2017.codfw.wmnet/dev/sda41.7T54%
restbase2017.codfw.wmnet/dev/sdb41.7T54%
restbase2017.codfw.wmnet/dev/sdc41.7T55%
restbase2018.codfw.wmnet/dev/sda41.7T47%
restbase2018.codfw.wmnet/dev/sdb41.7T49%
restbase2018.codfw.wmnet/dev/sdc41.7T53%
restbase2023.codfw.wmnet/dev/sda41.7T47%
restbase2023.codfw.wmnet/dev/sdb41.7T45%
restbase2023.codfw.wmnet/dev/sdc41.7T45%

We have hosts equipped with two different SSD configurations, one resulting in 4 @ 1.4T filesystems and one with 3 @ 1.7T, for totals of ~5.6 and ~5.1T respectively. We are however constrained by the least common denominator, or 5.1T/host. That leaves us currently at ~50% of total space, and a value that now agrees with the Grafana dashboard:

NOTE: It would be unwise to operate the cluster at anywhere near total disk capacity, so "50%" utilization should NOT be interpreted to mean we have another ~37T of usable capacity available.
Eevans claimed this task.