releases1001 has full / partition
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Feb 23 2018, 10:24 AM

Description

10:04:13 <icinga-wm> PROBLEM - Disk space on releases1001 is CRITICAL: DISK CRITICAL - free space: / 4716 MB (3% inode=74%)

/var/lib/jenkins is 99G, there are ~3G left in the root partition

17G	REL1_27
17G	REL1_29
17G	REL1_30
50G	jobs

From Elukey: /var/lib/jenkins/jobs/mediawiki-private-nightlies/workspace/BRANCH seems the culprit

Actions to do:

Make sure jobs clean up their mess after completion
Migrate /var/lib/jenkins/ to a dedicated partition not shared with /

Event Timeline

hashar created this task.Feb 23 2018, 10:24 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 23 2018, 10:24 AM

Mentioned in SAL (#wikimedia-operations) [2018-02-23T10:30:00Z] <hashar> releases1001: sudo -u jenkins rm -fR /var/lib/jenkins/jobs/mediawiki-private-nightlies/workspace/BRANCH/REL1_??/mediawiki-snapshot-REL1_??-2018???? # T188080

I have deleted the mediawiki snapshots directories from the jobs. They are kept behind on each runs. Surely they should be deleted once the tarball has been created and the tarball should be archived outside of the workspace (if not already) and deleted.

I'm redoing the tarball generation process anyway, might as well just disable the jobs too.

The immediate issue was resolved. Future work will be tracked separately.

releases1001 has full / partitionClosed, ResolvedPublicActions

Description

Event Timeline

releases1001 has full / partition
Closed, ResolvedPublic
Actions