Change Details

From IRC: ``` [06:20:22] <+icinga-wm> PROBLEM - Disk space on releases1002 is CRITICAL: DISK CRITICAL - free space: /srv/docker 1023 MB (0% inode=65%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=releases1002&var-datasource=eqiad+prometheus/ops ``` And it is indeed full: ``` root@releases1002:~# df -hT /srv/docker Filesystem Type Size Used Avail Use% Mounted on /dev/vdb1 ext4 147G 139G 1.0G 100% /srv/docker ```

From IRC: ``` [06:20:22] <+icinga-wm> PROBLEM - Disk space on releases1002 is CRITICAL: DISK CRITICAL - free space: /srv/docker 1023 MB (0% inode=65%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=releases1002&var-datasource=eqiad+prometheus/ops ``` And it is indeed full: ``` root@releases1002:~# df -hT /srv/docker Filesystem Type Size Used Avail Use% Mounted on /dev/vdb1 ext4 147G 139G 1.0G 100% /srv/docker ``` The `/srv/docker` partition is routinely filling up. The reason is some Jenkins job to build MediaWiki images which do not prune intermediate containers / images: {F34575487 size=full} That got manually cleaned up today 8/4 and had to be cleaned up last week as well: ``` lang=irc 2021-07-30 21:27 <dduvall> "Total reclaimed space: 141.4GB" on releases1002 following docker prune ```

From IRC: ``` [06:20:22] <+icinga-wm> PROBLEM - Disk space on releases1002 is CRITICAL: DISK CRITICAL - free space: /srv/docker 1023 MB (0% inode=65%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=releases1002&var-datasource=eqiad+prometheus/ops ``` And it is indeed full: ``` root@releases1002:~# df -hT /srv/docker Filesystem Type Size Used Avail Use% Mounted on /dev/vdb1 ext4 147G 139G 1.0G 100% /srv/docker ``` The `/srv/docker` partition is routinely filling up. The reason is some Jenkins job to build MediaWiki images which do not prune intermediate containers / images: {F34575487 size=full} That got manually cleaned up today 8/4 and had to be cleaned up last week as well: ``` lang=irc 2021-07-30 21:27 <dduvall> "Total reclaimed space: 141.4GB" on releases1002 following docker prune ```