integration-slave-jessie-XXXX is out of disk space
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/php55lint/56708/console

06:48:49  > git checkout -f 8e2a2daa134f73f7468d791c4c660ec735054842
06:48:49 FATAL: Could not checkout 8e2a2daa134f73f7468d791c4c660ec735054842
06:48:49 hudson.plugins.git.GitException: Command "git checkout -f 8e2a2daa134f73f7468d791c4c660ec735054842" returned status code 128:
06:48:49 stdout: 
06:48:49 stderr: error: unable to write file modules/wikivoyage/i18n/en.json
...
06:48:49 fatal: cannot create directory at 'modules/wikivoyage/images': No space left on device
Catrope created this task.Mar 10 2018, 6:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2018, 6:51 AM

Mentioned in SAL (#wikimedia-releng) [2018-03-10T07:03:09Z] <greg-g> gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - T189365

greg closed this task as Resolved.Mar 10 2018, 7:03 AM
greg claimed this task.
greg added a subscriber: greg.

Should be fixed next run.

Restricted Application added a project: User-greg. · View Herald TranscriptMar 10 2018, 7:03 AM

Change 419361 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate phpunit-coverage-patch jobs to Nodepool

https://gerrit.wikimedia.org/r/419361

That keeps happening, those slaves are kept between builds and the build workspaces keep growing up. There are a few potential fixes:

Mentioned in SAL (#wikimedia-releng) [2018-03-14T08:05:27Z] <hashar> Move Jenkins jobs mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch to Nodepool (they use too much disk space) | T189365

hashar renamed this task from integration-slave-jessie-1004 is out of disk space to integration-slave-jessie-XXXX is out of disk space.Mar 14 2018, 8:06 AM

Change 419361 merged by jenkins-bot:
[integration/config@master] Migrate phpunit-coverage-patch jobs to Nodepool

https://gerrit.wikimedia.org/r/419361

Something's still wrong with integration-slave-jessie-1001:

https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1026/console

08:59:42 + echo 'Got exit code 1 from PY_COLORS=1 tox -v'
08:59:42 + tox --version
08:59:42 ++ ' >> log/phab_comment.log
08:59:42         tail -10 log/stdout.log >> log/phab_comment.log
08:59:42         echo '
08:59:42 /tmp/jenkins8105632489456413334.sh: line 28:  >> log/phab_comment.log
08:59:42         tail -10 log/stdout.log >> log/phab_comment.log
08:59:42         echo : No such file or directory
08:59:42 + echo ''
08:59:42 + exit 1
08:59:42 Build step 'Execute shell' marked build as failure

@Gilles Indeed though that error is unrelated to disk space. That specific slave has a lot of disk space available. Can you fill it as another task and subscribe @mmodell . Thanks!

greg removed greg as the assignee of this task.Mar 19 2018, 9:57 PM
greg removed a project: User-greg.
hashar closed this task as Resolved.Mar 20 2018, 9:48 AM
hashar claimed this task.

Solved by migrating to Nodepool the jobs that generate MediaWiki coverage report per patch