Page MenuHomePhabricator
Create Task
Maniphest T243223

Occasional Selenium build failures related to temporary directories / no space left
Closed, ResolvedPublic
Actions

Description

Excerpts from some failed builds, all on integration-agent-docker-1008:

https://integration.wikimedia.org/ci/job/quibble-vendor-selenium-docker/2644/consoleFull:

13:44:14 INFO:backend.DevWebServer:[Mon Jan 20 12:44:14 2020] PHP Notice:  tempnam(): file created in the system's temporary directory in /workspace/src/vendor/zordius/lightncandy/src/LightnCandy.php on line 152
13:44:14 INFO:backend.DevWebServer:[Mon Jan 20 12:44:14 2020] Can not generate tmp file under /tmp!!
13:44:14 INFO:backend.DevWebServer:
13:44:14 INFO:backend.DevWebServer:[Mon Jan 20 12:44:14 2020] PHP Notice:  tempnam(): file created in the system's temporary directory in /workspace/src/vendor/zordius/lightncandy/src/LightnCandy.php on line 152
13:44:14 INFO:backend.DevWebServer:[Mon Jan 20 12:44:14 2020] Can not generate tmp file under /tmp!!
...
13:44:15 ERROR: unknown error: cannot create temp dir for user data dir
13:44:15   (Driver info: chromedriver=73.0.3683.75,platform=Linux 4.9.0-11-amd64 x86_64)
13:44:15 chrome

https://integration.wikimedia.org/ci/job/quibble-vendor-selenium-docker/2675/consoleFull:

17:09:39 INFO:backend.DevWebServer:[Mon Jan 20 16:09:39 2020] PHP Notice:  tempnam(): file created in the system's temporary directory in /workspace/src/vendor/zordius/lightncandy/src/LightnCandy.php on line 152
17:09:39 INFO:backend.DevWebServer:[Mon Jan 20 16:09:39 2020] Can not generate tmp file under /tmp!!
17:09:39 INFO:backend.DevWebServer:
17:09:39 INFO:backend.DevWebServer:[Mon Jan 20 16:09:39 2020] PHP Notice:  tempnam(): file created in the system's temporary directory in /workspace/src/vendor/zordius/lightncandy/src/LightnCandy.php on line 152
17:09:39 INFO:backend.DevWebServer:[Mon Jan 20 16:09:39 2020] Can not generate tmp file under /tmp!!
17:09:39 INFO:backend.DevWebServer:
17:09:39 INFO:backend.DevWebServer:[Mon Jan 20 16:09:39 2020] 127.0.0.1:35126 [500]: //index.php?title=Special%3AUserLogin
...
17:09:39 INFO:backend.ChromeWebDriver:Terminating ChromeWebDriver
17:09:39 INFO:backend.Xvfb:Terminating Xvfb
17:09:39 The XKEYBOARD keymap compiler (xkbcomp) reports:
17:09:39 > Error:            Cannot open "/tmp/server-94.xkm" to write keyboard description
17:09:39 >                   Exiting
17:09:39 The XKEYBOARD keymap compiler (xkbcomp) reports:
17:09:39 > Error:            Cannot open "/tmp/server-94.xkm" to write keyboard description
17:09:39 >                   Exiting
17:09:39 XKB: Failed to compile keymap
17:09:39 Keyboard initialization failed. This could be a missing or incorrect setup of xkeyboard-config.
17:09:39 (EE) 
17:09:39 Fatal server error:
17:09:39 (EE) Failed to activate virtual core keyboard: 2(EE)

https://integration.wikimedia.org/ci/job/quibble-vendor-selenium-docker/2677/consoleFull:

17:09:41  = [up to date]            1.31.1                 -> 1.31.1
17:09:41  = [up to date]            1.31.2                 -> 1.31.2
17:09:41  = [up to date]            1.31.3                 -> 1.31.3
17:09:41 error: cannot lock ref 'refs/tags/1.31.4': Unable to create '/workspace/src/.git/refs/tags/1.31.4.lock': No space left on device
17:09:41  ! [new tag]               1.31.4                 -> 1.31.4  (unable to update local ref)
17:09:41 error: cannot lock ref 'refs/tags/1.31.5': Unable to create '/workspace/src/.git/refs/tags/1.31.5.lock': No space left on device
17:09:41  ! [new tag]               1.31.5                 -> 1.31.5  (unable to update local ref)
17:09:41 error: cannot lock ref 'refs/tags/1.31.6': Unable to create '/workspace/src/.git/refs/tags/1.31.6.lock': No space left on device
17:09:41  ! [new tag]               1.31.6                 -> 1.31.6  (unable to update local ref)

Event Timeline

Lucas_Werkmeister_WMDE created this task.Jan 20 2020, 4:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 20 2020, 4:37 PM
Lucas_Werkmeister_WMDE added a comment.Jan 20 2020, 4:39 PM

The file system should have plenty of disk space (checked by @Reedy):

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda2        19G  2.5G   16G  15% /
Lucas_Werkmeister_WMDE added a comment.Jan 20 2020, 4:40 PM

Not high priority for now, let’s see if this keeps happening I guess.

Lucas_Werkmeister_WMDE renamed this task from Occasional build failures related to temporary directories on integration-agent-docker-1008 to Occasional build failures related to temporary directories / no space left on integration-agent-docker-1008.Jan 20 2020, 4:47 PM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)
Lucas_Werkmeister_WMDE added a comment.EditedJan 24 2020, 2:58 PM

Now affecting integration-agent-docker-1002. (On the other hand, I don’t remember seeing this on integration-agent-docker-1008 very recently.)

https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/16965/consoleFull:

14:54:21 > Wikibase@0.1.0 install:tainted-ref /workspace/src/extensions/Wikibase
14:54:21 > npm --prefix view/lib/wikibase-tainted-ref ci
14:54:21 
14:54:44 WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/Wikibase/view/lib/wikibase-tainted-ref/node_modules/fbjs/node_modules/core-js/fn/string/includes.js'
14:54:44 WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/Wikibase/client/data-bridge/node_modules/@storybook/api/node_modules/core-js/modules/es.string.sup.js'
14:54:44 WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/Wikibase/view/lib/wikibase-tainted-ref/node_modules/@storybook/router/node_modules/core-js/stable/date/to-string.js'
14:54:44 WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/Wikibase/client/data-bridge/node_modules/@storybook/vue/node_modules/@webassemblyjs/helper-api-error/lib/index.js'
14:54:44 WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/Wikibase/client/data-bridge/node_modules/remark-mdx/node_modules/@babel/traverse/lib/path/removal.js'

https://integration.wikimedia.org/ci/job/quibble-vendor-selenium-docker/3035/consoleFull:

14:54:41 > @ selenium-test /workspace/src/extensions/WikibaseLexeme
14:54:41 > wdio tests/selenium/wdio.conf.js
14:54:41 
14:54:44 ERROR: unknown error: Chrome failed to start: exited abnormally
14:54:44   (unknown error: DevToolsActivePort file doesn't exist)
14:54:44   (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
14:54:44   (Driver info: chromedriver=73.0.3683.75,platform=Linux 4.9.0-11-amd64 x86_64)
14:54:44 chrome
14:54:44     at new RuntimeError (/workspace/src/extensions/WikibaseLexeme/node_modules/webdriverio/build/lib/utils/ErrorHandler.js:143:12)
14:54:44     at Request._callback (/workspace/src/extensions/WikibaseLexeme/node_modules/webdriverio/build/lib/utils/RequestHandler.js:316:39)
14:54:44     at Request.self.callback (/workspace/src/extensions/WikibaseLexeme/node_modules/request/request.js:185:22)
14:54:44     at Request.emit (events.js:189:13)
14:54:44     at Request.EventEmitter.emit (domain.js:441:20)
14:54:44     at Request.<anonymous> (/workspace/src/extensions/WikibaseLexeme/node_modules/request/request.js:1157:10)
14:54:44     at Request.emit (events.js:189:13)
14:54:44     at Request.EventEmitter.emit (domain.js:441:20)
14:54:44     at IncomingMessage.<anonymous> (/workspace/src/extensions/WikibaseLexeme/node_modules/request/request.js:1079:12)
14:54:44     at Object.onceWrapper (events.js:277:13)
14:54:44 
14:54:46 ERROR: unknown error: cannot create temp dir for user data dir
14:54:46   (Driver info: chromedriver=73.0.3683.75,platform=Linux 4.9.0-11-amd64 x86_64)
14:54:46 chrome
14:54:46     at new RuntimeError (/workspace/src/extensions/WikibaseLexeme/node_modules/webdriverio/build/lib/utils/ErrorHandler.js:143:12)
14:54:46     at Request._callback (/workspace/src/extensions/WikibaseLexeme/node_modules/webdriverio/build/lib/utils/RequestHandler.js:316:39)
14:54:46     at Request.self.callback (/workspace/src/extensions/WikibaseLexeme/node_modules/request/request.js:185:22)
14:54:46     at Request.emit (events.js:189:13)
14:54:46     at Request.EventEmitter.emit (domain.js:441:20)
14:54:46     at Request.<anonymous> (/workspace/src/extensions/WikibaseLexeme/node_modules/request/request.js:1157:10)
14:54:46     at Request.emit (events.js:189:13)
14:54:46     at Request.EventEmitter.emit (domain.js:441:20)
14:54:46     at IncomingMessage.<anonymous> (/workspace/src/extensions/WikibaseLexeme/node_modules/request/request.js:1079:12)
14:54:46     at Object.onceWrapper (events.js:277:13)
Lucas_Werkmeister_WMDE renamed this task from Occasional build failures related to temporary directories / no space left on integration-agent-docker-1008 to Occasional Selenium build failures related to temporary directories / no space left.Jan 24 2020, 3:01 PM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)
Lucas_Werkmeister_WMDE added a comment.EditedFeb 5 2020, 2:00 PM

Now on integration-agent-docker-1006.

https://integration.wikimedia.org/ci/job/quibble-vendor-selenium-docker/3677/consoleFull:

14:52:32 INFO:zuul.CloneMapper:Mapping projects to workspace...
14:52:32 INFO:zuul.CloneMapper:  mediawiki/extensions/MobileFrontend -> extensions/MobileFrontend
14:52:32 INFO:zuul.CloneMapper:Expansion completed.
14:52:32 INFO:quibble.commands:Running webdriver test in /workspace/src/extensions/MobileFrontend
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/less-5e48c73a/test/css/extend.css'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/stylelint-fb2ce29a/lib/rules/selector-max-empty-lines/README.md'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/@babel/types-0e51a09b/lib/utils/shallowEqual.js'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/core-js-83304a5e/modules/esnext.map.every.js'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/jsdom-6913cf8e/lib/jsdom/living/nodes/HTMLInputElement-impl.js'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/string.prototype.padstart-3ccbb44d/package.json'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/string.prototype.matchall-4e535d84/polyfill.js'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/internal-slot-7b5a8362/.github/FUNDING.yml'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/string.prototype.padend-6aaf5d02/package.json'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/object.values-8bb0c611/test/index.js'
14:52:44 npm WARN tar ENOSPC: no space left on device, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/promise.allsettled-1160ec82/test/index.js'
...
14:52:48 npm WARN tar ENOENT: no such file or directory, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/core-js-8e0be6c7/modules/web.immediate.js'
14:52:48 npm WARN tar ENOENT: no such file or directory, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/core-js-8e0be6c7/modules/web.queue-microtask.js'
14:52:48 npm WARN tar ENOENT: no such file or directory, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/core-js-8e0be6c7/modules/web.timers.js'
14:52:48 npm WARN tar ENOENT: no such file or directory, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/core-js-8e0be6c7/modules/web.url-search-params.js'
14:52:48 npm WARN tar ENOENT: no such file or directory, open '/workspace/src/extensions/MobileFrontend/node_modules/.staging/core-js-8e0be6c7/modules/web.url.js'
14:52:49 npm ERR! path /workspace/src/extensions/MobileFrontend/node_modules/.staging/string_decoder-f0a0daff
14:52:49 npm ERR! code ENOSPC
14:52:49 npm ERR! errno -28
14:52:49 npm ERR! syscall mkdir
14:52:49 npm ERR! nospc ENOSPC: no space left on device, mkdir '/workspace/src/extensions/MobileFrontend/node_modules/.staging/string_decoder-f0a0daff'
14:52:49 npm ERR! nospc There appears to be insufficient space on your system to finish.
14:52:49 npm ERR! nospc Clear up some disk space and try again.
Krinkle added a project: Wikimedia-production-error (ARCHIVED -- Shared Build Failure).Mar 20 2020, 5:51 PM
Jdforrester-WMF added subscribers: hashar, Jdforrester-WMF.Apr 3 2020, 4:39 PM

Per @hashar, it's not running on the / mount point but in another partition which might not have as much space?

Lucas_Werkmeister_WMDE added a comment.Apr 3 2020, 4:40 PM

(For what it’s worth, I don’t remember encountering this error for quite a while now.)

Jdforrester-WMF closed this task as Resolved.Apr 8 2020, 6:41 PM

Optimistically declaring Resolved, then.

hashar added a comment.Aug 6 2020, 2:22 PM

No idea what happened in January 2020. I guess some cache got oversized or the partition holding the Jenkins workspaces ended up filed somehow. That was definitely an infrastructure issue.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL