Page MenuHomePhabricator

releases1001 has full / partition
Closed, ResolvedPublic

Description

10:04:13 <icinga-wm> PROBLEM - Disk space on releases1001 is CRITICAL: DISK CRITICAL - free space: / 4716 MB (3% inode=74%)

/var/lib/jenkins is 99G, there are ~3G left in the root partition

17G	REL1_27
17G	REL1_29
17G	REL1_30
50G	jobs

From Elukey: /var/lib/jenkins/jobs/mediawiki-private-nightlies/workspace/BRANCH seems the culprit

Actions to do:

  • Make sure jobs clean up their mess after completion
  • Migrate /var/lib/jenkins/ to a dedicated partition not shared with /

Event Timeline

hashar created this task.Feb 23 2018, 10:24 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 23 2018, 10:24 AM

Mentioned in SAL (#wikimedia-operations) [2018-02-23T10:30:00Z] <hashar> releases1001: sudo -u jenkins rm -fR /var/lib/jenkins/jobs/mediawiki-private-nightlies/workspace/BRANCH/REL1_??/mediawiki-snapshot-REL1_??-2018???? # T188080

I have deleted the mediawiki snapshots directories from the jobs. They are kept behind on each runs. Surely they should be deleted once the tarball has been created and the tarball should be archived outside of the workspace (if not already) and deleted.

demon added a comment.Feb 23 2018, 8:58 PM

I'm redoing the tarball generation process anyway, might as well just disable the jobs too.

greg closed this task as Resolved.Jul 13 2018, 10:18 PM
greg assigned this task to hashar.
greg added a subscriber: greg.

The immediate issue was resolved. Future work will be tracked separately.