Hi, noticed that the math19.math.eqiad1.wikimedia.cloud Cloud VPS instance has filled its disks.
root@math19:~# df -h / /srv Filesystem Size Used Avail Use% Mounted on /dev/vda2 19G 19G 0 100% / /dev/mapper/vd-second--local--disk 60G 60G 0 100% /srv
Please free up some disk space so that Puppet runs will continue to run properly. Thanks in advance!
this is tricky, the volume is completely occupied by docker, and I can not start docker as there is no space left
I am not sure about the specifics on this deployment but maybe its worth checking docker prune to cleanup old unused images/containers:
https://docs.docker.com/engine/reference/commandline/system_prune/
The command only works if docker is running. So I manually deleted some old logs. Thereafter I could start docker and eventually run the prune command.