Page MenuHomePhabricator

wmf-quibble-vendor-mysql-php74-docker failing due to insufficient space
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php74-docker/23630/consoleFull

10:29:28 npm WARN tar TAR_ENTRY_ERROR ENOENT: no such file or directory, lstat '/workspace/src/node_modules/rxjs/src/internal/operators'
10:29:28 npm WARN tar TAR_ENTRY_ERROR ENOENT: no such file or directory, lstat '/workspace/src/node_modules/rxjs/dist/types'
10:29:28 npm WARN tar TAR_ENTRY_ERROR ENOENT: no such file or directory, lstat '/workspace/src/node_modules/rxjs/src/internal/operators'
10:29:28 npm WARN tar TAR_ENTRY_ERROR ENOENT: no such file or directory, lstat '/workspace/src/node_modules/rxjs/dist/types'
10:29:28 npm WARN tar TAR_ENTRY_ERROR ENOENT: no such file or directory, lstat '/workspace/src/node_modules/rxjs/src/internal/observable'
10:29:28 npm WARN tar TAR_ENTRY_ERROR ENOENT: no such file or directory, lstat '/workspace/src/node_modules/rxjs/src/internal/operators'
10:29:28 npm WARN tar TAR_ENTRY_ERROR ENOENT: no such file or directory, lstat '/workspace/src/node_modules/rxjs/src/internal/operators'
10:29:28 npm WARN tar TAR_ENTRY_ERROR ENOENT: no such file or directory, lstat '/workspace/src/node_modules/rxjs/src/internal/operators'
10:29:28 npm ERR! code ENOSPC
10:29:28 npm ERR! syscall mkdir
10:29:28 npm ERR! path /cache/npm/_cacache/content-v2/sha512/59
10:29:28 npm ERR! errno -28
10:29:28 npm ERR! nospc ENOSPC: no space left on device, mkdir '/cache/npm/_cacache/content-v2/sha512/59'
10:29:28 npm ERR! nospc There appears to be insufficient space on your system to finish.
10:29:28 npm ERR! nospc Clear up some disk space and try again.
10:29:28 
10:29:28 npm ERR! A complete log of this run can be found in:
10:29:28 npm ERR!     /cache/npm/_logs/2023-05-17T09_29_27_976Z-debug.log
10:29:28 <<< Finish: npm install in /workspace/src, in 41.673 s

Event Timeline

Mentioned in SAL (#wikimedia-releng) [2023-05-17T09:53:35Z] <hashar> integration-agent-docker-1029: erased a few files under /srv. I suspect the 36GBytes partition is not large enough nowadays for multiple concurrent builds of MediaWiki # T336840

hashar claimed this task.
hashar subscribed.

I think that is because the /srv partition is 36 GBytes which is a little short when there is left over materials from previously build and 3 concurrent MediaWiki jobs run on the same host. Most probably we would need larger instances. I think I will look at getting large instance as part of the Debian BullseyeBookworm switch. And we might move the them to use a large tmpfs instead of disk.