Page MenuHomePhabricator

OSError: [Errno 28] No space left on device on compiler02.puppet3-diffs.eqiad.wmflabs
Closed, ResolvedPublic

Description

There seems to be a disk space issue on compiler02.puppet3-diffs.eqiad.wmflabs.

I got the following error running utils/pcc:

$ ./utils/pcc 306183 cp1008.wikimedia.org
Compiling 306183 on node(s) cp1008.wikimedia.org...
Your build URL is https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/3802
Started by user ema
Building remotely on compiler02.puppet3-diffs.eqiad.wmflabs in workspace /mnt/jenkins-workspace/workspace/operations-puppet-catalog-compiler
[operations-puppet-catalog-compiler] $ /bin/bash -xe /tmp/hudson5550531486441557491.sh
+ CHANGE=306183
+ NODES=cp1008.wikimedia.org
+ puppet-compiler
[ 2016-08-23T13:31:45 ] INFO: Working on change 306183
[ 2016-08-23T13:31:45 ] INFO: Refreshing the common repos from upstream if needed
[ 2016-08-23T13:31:46 ] INFO: Creating directories under /mnt/jenkins-workspace/puppet-compiler
[ 2016-08-23T13:31:46 ] CRITICAL: Build run failed: [Errno 28] No space left on device: '/mnt/jenkins-workspace/puppet-compiler/3802'
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/puppet_compiler/cli.py", line 41, in main
success = c.run()
File "build/bdist.linux-x86_64/egg/puppet_compiler/controller.py", line 91, in run
self.m.prepare()
File "build/bdist.linux-x86_64/egg/puppet_compiler/prepare.py", line 51, in prepare
os.mkdir(self.base_dir, 0755)
OSError: [Errno 28] No space left on device: '/mnt/jenkins-workspace/puppet-compiler/3802'
Build step 'Execute shell' marked build as failure
Finished:  FAIL

Event Timeline

Mentioned in SAL [2016-08-23T13:37:12Z] <godog> remove some old compilations from puppet compiler filling up compiler02.eqiad.wmflabs disks T143671

greg triaged this task as High priority.Aug 23 2016, 4:41 PM
greg added subscribers: fgiunchedi, greg.

(Set it to High, but I assume @fgiunchedi fixed it by removing the old compilations.)

Change 306399 had a related patch set uploaded (by Filippo Giunchedi):
puppet_compiler: clean older output dirs

https://gerrit.wikimedia.org/r/306399

yup @greg ! I've proposed a patch to cleanup old compilation dirs, though in this case there were big compilations taking up a lot of space which wouldn't have been cleaned up anyways (not old enough)

Change 306399 merged by Filippo Giunchedi:
puppet_compiler: clean older output dirs

https://gerrit.wikimedia.org/r/306399