This job started failing today: https://integration.wikimedia.org/ci/job/phabricator-jessie-commits/
And the failure points to a full disk:
13:08:46 Exception: 13:08:46 Traceback (most recent call last): 13:08:46 File "/srv/jenkins-workspace/workspace/phabricator-jessie-commits/source/.tox/py27/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main 13:08:46 status = self.run(options, args) 13:08:46 File "/srv/jenkins-workstee: log/stdout.log: No space left on device
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| Migrate phpunit-coverage-patch jobs to Nodepool | Hashar | integration/config | master | +3 -4 |
Mentioned in SAL (#wikimedia-operations) [2018-03-13T17:06:37Z] <godog> cleanup integration-slave-jessie-1001:/srv/pbuilder/build - T189587
Would be helpful if we had a cleanup policy for artifacts so operators don't have to manually chase and delete things to recover disk space.
The biggest dir in /srv/jenkins-workspace/workspace took almost 5G (apps-android-wikipedia-test), while the whole /srv disk is only allocated 21G.
OK, jessie-1002 and jessie-1001 are cleaned up. Was getting some untar errors on docker-1002 too, so I'll see if I can clean that up.
docker-1002 now also cleaned up, but also had the 5G android test dir on a 21G disk. Seems like the out-of-space errors kicked in at 3G free too, so maybe we could dial down the reserved space from 15%.
Change 419361 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate phpunit-coverage-patch jobs to Nodepool
Change 419361 merged by jenkins-bot:
[integration/config@master] Migrate phpunit-coverage-patch jobs to Nodepool