I've cleaned outputs older than 31 days, that gave us almost 5G:

root@compiler1002:/srv/jenkins-workspace/puppet-compiler/output# find ./ -type d -ctime +31 -maxdepth 1 -exec rm -rf {} +
root@compiler1002:/srv/jenkins-workspace/puppet-compiler/output# df -h
Filesystem                          Size  Used Avail Use% Mounted on
udev                                3.9G     0  3.9G   0% /dev
tmpfs                               799M   83M  717M  11% /run
/dev/vda3                            19G  3.2G   15G  19% /
tmpfs                               4.0G  4.0K  4.0G   1% /dev/shm
tmpfs                               5.0M     0  5.0M   0% /run/lock
tmpfs                               4.0G     0  4.0G   0% /sys/fs/cgroup
/dev/mapper/vd-second--local--disk   60G   52G  4.8G  92% /srv

Change 507623 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] puppet_compiler: add cron to delete old output files

https://gerrit.wikimedia.org/r/507623

In T222072#5146890, @hashar wrote:

• add a puppet tidy class to purge old artifacts

I think it can be even simpler. How about https://gerrit.wikimedia.org/r/c/operations/puppet/+/507623/2/modules/profile/manifests/puppet_compiler.pp

In T222072#5150744, @Dzahn wrote:

I think it can be even simpler. How about https://gerrit.wikimedia.org/r/c/operations/puppet/+/507623/2/modules/profile/manifests/puppet_compiler.pp

I agree and think this is a good step to address this.

In addition I think we'll want to implement compression as @hashar suggests, and/or deploy larger /srv filesystems on the worker hosts. Since compiling for a large number of hosts can create output directories up to ~1G in size, and enough of these could be created within a 30 day window to fill the filesystem.

On paper this use case also would lend itself to a filesystem with transparent compression. Maybe btrfs with compression. The data stored on disk is non-critical, and there are multiple worker nodes should issues arise with one filesystem.

Change 507623 merged by Dzahn:
[operations/puppet@production] puppet_compiler: add cron to delete old output files

https://gerrit.wikimedia.org/r/507623

cron deployed on compiler1002 and tested to run manually. works. removed about 2G (from 44G to 42G size of the output dir).

is there more to the ticket now?

• to have the CI job or puppet-compiler to gzip those .pson files
• add a puppet tidy class to purge old artifacts

So either have the Jenkins jobs to compress artifacts or have puppet-compiler to do it.