Page MenuHomePhabricator

Cassandra Grafana dashboards seem to disagree with actual utilization
Open, MediumPublic

Description

Current storage utilization via Grafana would suggest ~30T of utilization out of ~76.5 total space, or ~39%. Actual disk utilization though seems to range from 60-78%.

eevans@hermes:$ cdsh -d eqiad -- "df -h |grep /srv/sd"
restbase1016.eqiad.wmnet: /dev/sdc4       1.4T  872G  548G  62% /srv/sdc4
restbase1016.eqiad.wmnet: /dev/sda4       1.4T  827G  594G  59% /srv/sda4
restbase1016.eqiad.wmnet: /dev/sdd4       1.4T  901G  519G  64% /srv/sdd4
restbase1016.eqiad.wmnet: /dev/sdb4       1.4T  850G  571G  60% /srv/sdb4
restbase1019.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  634G  63% /srv/sdb4
restbase1019.eqiad.wmnet: /dev/sda4       1.7T  1.1T  565G  67% /srv/sda4
restbase1019.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  553G  68% /srv/sdc4
restbase1020.eqiad.wmnet: /dev/sda4       1.7T  1.2T  468G  73% /srv/sda4
restbase1020.eqiad.wmnet: /dev/sdb4       1.7T  1.2T  485G  72% /srv/sdb4
restbase1020.eqiad.wmnet: /dev/sdc4       1.7T  1.3T  388G  78% /srv/sdc4
restbase1021.eqiad.wmnet: /dev/sda4       1.7T  1.1T  638G  63% /srv/sda4
restbase1021.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  578G  66% /srv/sdb4
restbase1021.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  516G  70% /srv/sdc4
restbase1017.eqiad.wmnet: /dev/sda4       1.4T  824G  596G  59% /srv/sda4
restbase1017.eqiad.wmnet: /dev/sdb4       1.4T  808G  612G  57% /srv/sdb4
restbase1017.eqiad.wmnet: /dev/sdc4       1.4T  830G  590G  59% /srv/sdc4
restbase1017.eqiad.wmnet: /dev/sdd4       1.4T  822G  598G  58% /srv/sdd4
restbase1022.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  527G  69% /srv/sdc4
restbase1022.eqiad.wmnet: /dev/sda4       1.7T  1.2T  510G  70% /srv/sda4
restbase1022.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  567G  67% /srv/sdb4
restbase1023.eqiad.wmnet: /dev/sdb4       1.7T  1.2T  539G  69% /srv/sdb4
restbase1023.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  546G  68% /srv/sdc4
restbase1023.eqiad.wmnet: /dev/sda4       1.7T  1.2T  482G  72% /srv/sda4
restbase1024.eqiad.wmnet: /dev/sda4       1.7T  1.1T  614G  64% /srv/sda4
restbase1024.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  615G  64% /srv/sdb4
restbase1024.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  540G  69% /srv/sdc4
restbase1018.eqiad.wmnet: /dev/sda4       1.4T  761G  659G  54% /srv/sda4
restbase1018.eqiad.wmnet: /dev/sdb4       1.4T  811G  610G  58% /srv/sdb4
restbase1018.eqiad.wmnet: /dev/sdc4       1.4T  773G  647G  55% /srv/sdc4
restbase1018.eqiad.wmnet: /dev/sdd4       1.4T  771G  649G  55% /srv/sdd4
restbase1025.eqiad.wmnet: /dev/sdc4       1.7T  1.1T  573G  67% /srv/sdc4
restbase1025.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  573G  67% /srv/sdb4
restbase1025.eqiad.wmnet: /dev/sda4       1.7T  1.1T  572G  67% /srv/sda4
restbase1026.eqiad.wmnet: /dev/sda4       1.7T  1.2T  510G  70% /srv/sda4
restbase1026.eqiad.wmnet: /dev/sdb4       1.7T  1.2T  494G  71% /srv/sdb4
restbase1026.eqiad.wmnet: /dev/sdc4       1.7T  1.3T  450G  74% /srv/sdc4
restbase1027.eqiad.wmnet: /dev/sdb4       1.7T  1.1T  579G  66% /srv/sdb4
restbase1027.eqiad.wmnet: /dev/sda4       1.7T  1.1T  587G  66% /srv/sda4
restbase1027.eqiad.wmnet: /dev/sdc4       1.7T  1.2T  555G  68% /srv/sdc4
eevans@hermes:$

See: T256863: restbase2009 down

Related Objects

Event Timeline

Eevans created this task.Mon, Jul 20, 4:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Jul 20, 4:12 PM
eprodromou triaged this task as Medium priority.Tue, Jul 21, 8:17 PM
fgiunchedi moved this task from Inbox to Radar on the observability board.Mon, Jul 27, 11:45 AM