Page MenuHomePhabricator

releases1001 has full / partition
Closed, ResolvedPublic


10:04:13 <icinga-wm> PROBLEM - Disk space on releases1001 is CRITICAL: DISK CRITICAL - free space: / 4716 MB (3% inode=74%)

/var/lib/jenkins is 99G, there are ~3G left in the root partition

17G	REL1_27
17G	REL1_29
17G	REL1_30
50G	jobs

From Elukey: /var/lib/jenkins/jobs/mediawiki-private-nightlies/workspace/BRANCH seems the culprit

Actions to do:

  • Make sure jobs clean up their mess after completion
  • Migrate /var/lib/jenkins/ to a dedicated partition not shared with /

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2018-02-23T10:30:00Z] <hashar> releases1001: sudo -u jenkins rm -fR /var/lib/jenkins/jobs/mediawiki-private-nightlies/workspace/BRANCH/REL1_??/mediawiki-snapshot-REL1_??-2018???? # T188080

I have deleted the mediawiki snapshots directories from the jobs. They are kept behind on each runs. Surely they should be deleted once the tarball has been created and the tarball should be archived outside of the workspace (if not already) and deleted.

I'm redoing the tarball generation process anyway, might as well just disable the jobs too.

greg assigned this task to hashar.
greg added a subscriber: greg.

The immediate issue was resolved. Future work will be tracked separately.