integration-slave-jessie-XXXX is out of disk space
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Catrope
	Mar 10 2018, 6:51 AM

Description

https://integration.wikimedia.org/ci/job/php55lint/56708/console

06:48:49  > git checkout -f 8e2a2daa134f73f7468d791c4c660ec735054842
06:48:49 FATAL: Could not checkout 8e2a2daa134f73f7468d791c4c660ec735054842
06:48:49 hudson.plugins.git.GitException: Command "git checkout -f 8e2a2daa134f73f7468d791c4c660ec735054842" returned status code 128:
06:48:49 stdout: 
06:48:49 stderr: error: unable to write file modules/wikivoyage/i18n/en.json
...
06:48:49 fatal: cannot create directory at 'modules/wikivoyage/images': No space left on device

Details

	Subject	Repo	Branch	Lines +/-
	Migrate phpunit-coverage-patch jobs to Nodepool	integration/config	master	+3 -4

Customize query in gerrit

Related Objects

Mentioned In: T189411: Jenkins randomly fails tests for AbuseFilter
Mentioned Here: T179963: Workspaces for mwgate-php55lint / mwgate-php70lint are getting huge
T186489: Move debian-glue jobs to Docker using Stretch as a base image

Event Timeline

Catrope created this task.Mar 10 2018, 6:51 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2018, 6:51 AM

Mentioned in SAL (#wikimedia-releng) [2018-03-10T07:03:09Z] <greg-g> gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - T189365

Should be fixed next run.

Restricted Application added a project: User-greg. · View Herald TranscriptMar 10 2018, 7:03 AM

hashar reopened this task as Open.Mar 14 2018, 7:52 AM

hashar merged a task: T189587: integration-slave-jessie-1001 and integration-slave-jessie-1002 out of disk space.

hashar added subscribers: • Gilles, Ejegg, Legoktm and 9 others.

Change 419361 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate phpunit-coverage-patch jobs to Nodepool

https://gerrit.wikimedia.org/r/419361

gerritbot added a project: Patch-For-Review.Mar 14 2018, 8:02 AM

That keeps happening, those slaves are kept between builds and the build workspaces keep growing up. There are a few potential fixes:

T179963: Workspaces for mwgate-php55lint / mwgate-php70lint are getting huge
Jobs using mediawiki/core + extensions wich are just too large ( mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch )
- https://gerrit.wikimedia.org/r/419361 Migrate phpunit-coverage-patch jobs to Nodepool
apps-android-wikipedia-test which keeps build cache and never garbage collect it
The Debian package jobs that can surely get moved to different slaves T186489: Move debian-glue jobs to Docker using Stretch as a base image

Mentioned in SAL (#wikimedia-releng) [2018-03-14T08:05:27Z] <hashar> Move Jenkins jobs mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch to Nodepool (they use too much disk space) | T189365

hashar renamed this task from integration-slave-jessie-1004 is out of disk space to integration-slave-jessie-XXXX is out of disk space.Mar 14 2018, 8:06 AM

Change 419361 merged by jenkins-bot:
[integration/config@master] Migrate phpunit-coverage-patch jobs to Nodepool

https://gerrit.wikimedia.org/r/419361

Something's still wrong with integration-slave-jessie-1001:

https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1026/console

08:59:42 + echo 'Got exit code 1 from PY_COLORS=1 tox -v'
08:59:42 + tox --version
08:59:42 ++ ' >> log/phab_comment.log
08:59:42         tail -10 log/stdout.log >> log/phab_comment.log
08:59:42         echo '
08:59:42 /tmp/jenkins8105632489456413334.sh: line 28:  >> log/phab_comment.log
08:59:42         tail -10 log/stdout.log >> log/phab_comment.log
08:59:42         echo : No such file or directory
08:59:42 + echo ''
08:59:42 + exit 1
08:59:42 Build step 'Execute shell' marked build as failure

@Gilles Indeed though that error is unrelated to disk space. That specific slave has a lot of disk space available. Can you fill it as another task and subscribe @mmodell . Thanks!

hashar removed a subtask: T189660: phabricator-jessie-diffs fails with "echo : No such file or directory".Mar 14 2018, 11:22 AM

hashar mentioned this in T189411: Jenkins randomly fails tests for AbuseFilter.Mar 16 2018, 8:48 AM

greg removed greg as the assignee of this task.Mar 19 2018, 9:57 PM

greg removed a project: User-greg.