Docker vfs is using most of the space (and should be replace by a proper storage driver):

sudo du -sh /srv/*
4.0K    /srv/deployment
1.2T    /srv/docker
143G    /srv/jenkins
1.5G    /srv/jenkins-agent
16K     /srv/lost+found

sudo du -h --max-depth=1 /srv/docker/
4.0K    /srv/docker/containers
16K     /srv/docker/tmp
72K     /srv/docker/network
28K     /srv/docker/volumes
319M    /srv/docker/buildkit
4.0K    /srv/docker/runtimes
4.0K    /srv/docker/swarm
1.2T    /srv/docker/vfs
58M     /srv/docker/image
8.0K    /srv/docker/plugins
1.2T    /srv/docker/

I'm unsure how to safely remove the data of old containers. But I'll try pruning some containers and images.

I ran sudo docker builder prune which freed most of the space in /srv:

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg0-srv   1.4T  194G  1.1T  16% /srv

The alert resolved.

I'll leave this task open to either replace vfs with a proper storage driver and/or tune the docker-system-prune-dangling.service and docker-system-prune-all.service jobs.

The disk usage is going up again

https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&from=now-6h&to=now&timezone=utc&var-server=contint1003&var-datasource=000000026&var-cluster=ci&refresh=5m&viewPanel=panel-28

In T432326#12127808, @Jelto wrote:

I'll leave this task open to either replace vfs with a proper storage driver

Yes please. Overlayfs is what is needed. This will require doing something like https://gerrit.wikimedia.org/r/c/operations/puppet/+/402797 for contint1003.

In T432326#12128622, @dancy wrote:

In T432326#12127808, @Jelto wrote:

I'll leave this task open to either replace vfs with a proper storage driver

Yes please. Overlayfs is what is needed. This will require doing something like https://gerrit.wikimedia.org/r/c/operations/puppet/+/402797 for contint1003.

I will take care of this.

Change #1311477 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] jenkins: enable overlay/overlayfs kernel modules for Docker

https://gerrit.wikimedia.org/r/1311477

Change #1311477 merged by Dzahn:

[operations/puppet@production] jenkins: enable overlay/overlayfs kernel modules for Docker

https://gerrit.wikimedia.org/r/1311477

thanks all!

patch has been deployed

[contint1003:~] $ lsmod | grep overlay
overlay               217088  0

Mentioned in SAL (#wikimedia-releng) [2026-07-16T16:33:03Z] <dancy> Restarted docker on contint1003 (T432326)

Mentioned in SAL (#wikimedia-releng) [2026-07-16T16:34:17Z] <dancy> Restarted docker on contint2003 (T432326)