Page MenuHomePhabricator

Disk space checks complaining on docker build hosts when building containers
Closed, ResolvedPublic

Description

From T172409

Found this ticket from the comment on Icinga when i noticed we have this alert now:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=copper&service=Disk+space

DISK CRITICAL - /var/lib/docker/containers/296af4efe8f6f57432c8905b9d09a558eca6ba4214f40b1183f1d7a794976745/shm is not accessible: Permission denied

Best course of actions is to probably just avoid checking these mount points.

Event Timeline

I believe this has already been resolved meanwhile. See the "-i ..docker" exclusions in these checks: ( --ignore-ereg-partition)

hieradata/role/common/ci/master.yaml:profile::base::check_disk_options: -w 10% -c 5% -W 6% -K 3% -l -e -A -i '/(var/lib|run)/docker/*' --exclude-type=tracefs
hieradata/role/common/builder.yaml:profile::base::check_disk_options: -w 10% -c 5% -W 6% -K 3% -l -e -A -i '/(var/lib|run)/docker/*' --exclude-type=tracefs
hieradata/role/common/kubernetes/worker.yaml:profile::base::check_disk_options: -w 10% -c 5% -W 6% -K 3% -l -e -A -i '/(var/lib|run)/(docker|kubelet)/*' --exclude-type=tracefs
hieradata/role/common/kubernetes/staging/worker.yaml:profile::base::check_disk_options: -w 10% -c 5% -W 6% -K 3% -l -e -A -i '/(var/lib|run)/(docker|kubelet)/*' --exclude-type=tracefs
Dzahn claimed this task.