Page MenuHomePhabricator

Chromium crashes in Jenkins because TMPDIR too long (Error: Socket path too long)
Closed, ResolvedPublic1 Estimated Story Points

Description

Last good build: https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-karma/1531/console
Started happening out of nowhere and consistently as of https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-karma/1532/console

00:00:41.806 INFO [launcher]: Starting browser Chrome
00:00:43.283 ERROR [launcher]: Cannot start Chrome
00:00:43.283 	
00:00:43.288 INFO [launcher]: Trying to start Chrome again (1/2).
00:00:44.298 ERROR [launcher]: Cannot start Chrome
00:00:44.298 	
00:00:44.302 INFO [launcher]: Trying to start Chrome again (2/2).
00:00:45.198 ERROR [launcher]: Cannot start Chrome
00:00:45.198

Event Timeline

Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.
Krinkle triaged this task as Unbreak Now! priority.
Krinkle updated the task description. (Show Details)
Krinkle set Security to None.
Krinkle added a subscriber: Jdforrester-WMF.

Change 198179 had a related patch set uploaded (by Krinkle):
[DON'T MERGE] Set karma logLevel to DEBUG

https://gerrit.wikimedia.org/r/198179

Manually running it on integration-slave1405 over ssh (steps) did not result in such failure, instead it passed fine.

While Karma has great log level controls. None of the log levels include anything from the browser process being launched. So even with https://gerrit.wikimedia.org/r/198179 applying logLevel=DEBUG, we only get:

https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-karma/1576/console

00:00:29.521 INFO [launcher]: Starting browser Chrome
00:00:29.521 DEBUG [temp-dir]: Creating temp dir at /mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1578/karma-56017644
00:00:29.523 DEBUG [launcher]: /usr/bin/chromium-browser --user-data-dir=/mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1578/karma-56017644 --no-default-browser-check --no-first-run --disable-default-apps --disable-popup-blocking --disable-translate http://localhost:9876/?id=56017644
00:00:29.538 DEBUG [watcher]: Resolved files:
00:00:29.538 	/mnt/jenkins-workspace/workspace/mediawiki-core-qunit-karma/src/node_modules/qunitjs/qunit/qunit.js
00:00:29.538 	/mnt/jenkins-workspace/workspace/mediawiki-core-qunit-karma/src/node_modules/karma-qunit/lib/adapter.js
00:00:30.299 DEBUG [launcher]: Process Chrome exited with code 0
00:00:30.299 ERROR [launcher]: Cannot start Chrome

I've dug into Karma itself and added a simple process stdout/sterr logging via Krinkle/karma@606aa2501a0, which exposes the actual problem:

https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-karma/1582/console

00:00:47.579 DEBUG [launcher]: Restarting Chrome
00:00:47.579 DEBUG [temp-dir]: Creating temp dir at /mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1583/karma-15636471
00:00:47.580 DEBUG [launcher]: /usr/bin/chromium-browser --user-data-dir=/mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1583/karma-15636471 --no-default-browser-check --no-first-run --disable-default-apps --disable-popup-blocking --disable-translate http://localhost:9876/?id=15636471
00:00:48.345 [k-launch] stderr: [17793:17793:0320/071732:FATAL:process_singleton_posix.cc(223)] Check failed: path.length() < arraysize(addr->sun_path). Socket path too long: /mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1583/.org.chromium.Chromium.X5MvEz/SingletonSocket
00:00:48.345 
00:00:48.408 DEBUG [launcher]: Process Chrome exited with code 0
00:00:48.408 ERROR [launcher]: Cannot start Chrome

"Socket path too long"
Upstream:
https://code.google.com/p/chromium/issues/detail?id=33097
https://code.google.com/p/chromium/issues/detail?id=469065

This was probably caused by either the switch from JOB_NAME to BUILD_TAG for tmpfs subdirectory (which includes the substring "jenkins" and pushed it over the limit). Or caused by the build number reaching 1000 and being one digit longer.

Either way, our tmp directories are too long for Linux socket and/or Chromium's use of it. The end result (our tmpfs mount + Jenkins directory + Karma directory + Chromium user data directory + Chromium socket file name) is 116 characters:

/mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1582/.org.chromium.Chromium.2t1TaP/SingletonSocket

Which is apparently too long..

Krinkle renamed this task from Jenkins cannot start Chrome to Chromium crashes in Jenkins because TMPDIR too long (Error: Socket path too long).Mar 20 2015, 7:40 AM

Change 198197 had a related patch set uploaded (by Krinkle):
Trim tmpfs path by using executor-number instead of build-tag

https://gerrit.wikimedia.org/r/198197

Change 198197 merged by jenkins-bot:
Use EXECUTOR_NUMBER instead of BUILD_TAG in TMPDIR

https://gerrit.wikimedia.org/r/198197

Change 198179 abandoned by Krinkle:
[DON'T MERGE] Set karma logLevel to DEBUG

https://gerrit.wikimedia.org/r/198179

Krinkle removed a project: Patch-For-Review.
Krinkle removed a subscriber: gerritbot.