The Jenkins agents building Debian packages ended up with full disk due to /srv/pbuilder/aptcache never being purged of old packages (T339171).
The agents are using Puppet module package_builder which I guess should delete old Debian packages from the aptcache either via a find or via whatever pbuilder command might exist for that.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| package_builder: autoclean apt cache | operations/puppet | production | +2 -0 |
pbuilder(8) has an option to clean it automatically:
--autocleanaptcache Clean apt cache automatically, to run `apt-get autoclean` to only keep the packages which are required for the version of Debian. This is useful when you keep a aptcache directory for each distribution and want to keep the size of the aptcache down.
That can be set via pbuilderrc(5):
AUTOCLEANAPTCACHE=yes Always run with --autocleanaptcache option.
And from apt-get(8):
autoclean (and the auto-clean alias since 1.1) Like clean, autoclean clears out the local repository of retrieved package files. The difference is that it only removes package files that can no longer be downloaded, and are largely useless. This allows a cache to be maintained over a long period without it growing out of control. ...
So we can set AUTOCLEANPACKAGE=yes in our pbuilderrc and that will take care of it ;)
Change 930653 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/puppet@production] package_builder: autoclean apt cache
Change 930653 merged by Muehlenhoff:
[operations/puppet@production] package_builder: autoclean apt cache
Should be good now. I have previously removed all caches from the CI instances so it is unlikely we can check the result of this change there. However on production build2001.codfw.wmnet that should be noticeable after someone has made a build.
After a quick check on integration-agent-pkgbuilder-1001 and integration-agent-pkgbuilder-1002 it looks like the old packages no more accumulate. I am assuming the same happens on the production build host and thus claiming this issue to be resolved.