Page MenuHomePhabricator
Create Task
Maniphest T93330

Chromium crashes in Jenkins because TMPDIR too long (Error: Socket path too long)
Closed, ResolvedPublic1 Estimated Story Points
Actions

Description

Last good build: https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-karma/1531/console
Started happening out of nowhere and consistently as of https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-karma/1532/console

00:00:41.806 INFO [launcher]: Starting browser Chrome
00:00:43.283 ERROR [launcher]: Cannot start Chrome
00:00:43.283 	
00:00:43.288 INFO [launcher]: Trying to start Chrome again (1/2).
00:00:44.298 ERROR [launcher]: Cannot start Chrome
00:00:44.298 	
00:00:44.302 INFO [launcher]: Trying to start Chrome again (2/2).
00:00:45.198 ERROR [launcher]: Cannot start Chrome
00:00:45.198

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Use EXECUTOR_NUMBER instead of BUILD_TAG in TMPDIRintegration/jenkinsmaster+3 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krinkle created this task.Mar 20 2015, 7:14 AM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added projects: Continuous-Integration-Infrastructure, Regression.
Krinkle subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2015, 7:14 AM
Krinkle claimed this task.Mar 20 2015, 7:15 AM
Krinkle triaged this task as Unbreak Now! priority.
Krinkle updated the task description. (Show Details)
Krinkle set Security to None.
Krinkle added a parent task: T74063: Jenkins: Convert mediawiki qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium).
Krinkle added a subscriber: Jdforrester-WMF.
gerritbot subscribed.Mar 20 2015, 7:16 AM

Change 198179 had a related patch set uploaded (by Krinkle):
[DON'T MERGE] Set karma logLevel to DEBUG

https://gerrit.wikimedia.org/r/198179

gerritbot added a project: Patch-For-Review.Mar 20 2015, 7:16 AM
Krinkle added a comment.EditedMar 20 2015, 7:25 AM

Manually running it on integration-slave1405 over ssh (steps) did not result in such failure, instead it passed fine.

While Karma has great log level controls. None of the log levels include anything from the browser process being launched. So even with https://gerrit.wikimedia.org/r/198179 applying logLevel=DEBUG, we only get:

https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-karma/1576/console

00:00:29.521 INFO [launcher]: Starting browser Chrome
00:00:29.521 DEBUG [temp-dir]: Creating temp dir at /mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1578/karma-56017644
00:00:29.523 DEBUG [launcher]: /usr/bin/chromium-browser --user-data-dir=/mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1578/karma-56017644 --no-default-browser-check --no-first-run --disable-default-apps --disable-popup-blocking --disable-translate http://localhost:9876/?id=56017644
00:00:29.538 DEBUG [watcher]: Resolved files:
00:00:29.538 	/mnt/jenkins-workspace/workspace/mediawiki-core-qunit-karma/src/node_modules/qunitjs/qunit/qunit.js
00:00:29.538 	/mnt/jenkins-workspace/workspace/mediawiki-core-qunit-karma/src/node_modules/karma-qunit/lib/adapter.js
00:00:30.299 DEBUG [launcher]: Process Chrome exited with code 0
00:00:30.299 ERROR [launcher]: Cannot start Chrome

I've dug into Karma itself and added a simple process stdout/sterr logging via Krinkle/karma@606aa2501a0, which exposes the actual problem:

https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-karma/1582/console

00:00:47.579 DEBUG [launcher]: Restarting Chrome
00:00:47.579 DEBUG [temp-dir]: Creating temp dir at /mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1583/karma-15636471
00:00:47.580 DEBUG [launcher]: /usr/bin/chromium-browser --user-data-dir=/mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1583/karma-15636471 --no-default-browser-check --no-first-run --disable-default-apps --disable-popup-blocking --disable-translate http://localhost:9876/?id=15636471
00:00:48.345 [k-launch] stderr: [17793:17793:0320/071732:FATAL:process_singleton_posix.cc(223)] Check failed: path.length() < arraysize(addr->sun_path). Socket path too long: /mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1583/.org.chromium.Chromium.X5MvEz/SingletonSocket
00:00:48.345 
00:00:48.408 DEBUG [launcher]: Process Chrome exited with code 0
00:00:48.408 ERROR [launcher]: Cannot start Chrome

"Socket path too long"
Upstream:
https://code.google.com/p/chromium/issues/detail?id=33097
https://code.google.com/p/chromium/issues/detail?id=469065

This was probably caused by either the switch from JOB_NAME to BUILD_TAG for tmpfs subdirectory (which includes the substring "jenkins" and pushed it over the limit). Or caused by the build number reaching 1000 and being one digit longer.

Either way, our tmp directories are too long for Linux socket and/or Chromium's use of it. The end result (our tmpfs mount + Jenkins directory + Karma directory + Chromium user data directory + Chromium socket file name) is 116 characters:

/mnt/home/jenkins-deploy/tmpfs/jenkins-mediawiki-core-qunit-karma-1582/.org.chromium.Chromium.2t1TaP/SingletonSocket

Which is apparently too long..

Krinkle renamed this task from Jenkins cannot start Chrome to Chromium crashes in Jenkins because TMPDIR too long (Error: Socket path too long).Mar 20 2015, 7:40 AM
gerritbot added a comment.Mar 20 2015, 7:43 AM

Change 198197 had a related patch set uploaded (by Krinkle):
Trim tmpfs path by using executor-number instead of build-tag

https://gerrit.wikimedia.org/r/198197

gerritbot added a comment.Mar 20 2015, 7:45 AM

Change 198197 merged by jenkins-bot:
Use EXECUTOR_NUMBER instead of BUILD_TAG in TMPDIR

https://gerrit.wikimedia.org/r/198197

gerritbot added a comment.Mar 20 2015, 7:56 AM

Change 198179 abandoned by Krinkle:
[DON'T MERGE] Set karma logLevel to DEBUG

https://gerrit.wikimedia.org/r/198179

Krinkle closed this task as Resolved.Mar 20 2015, 8:35 AM
Krinkle removed a project: Patch-For-Review.
Krinkle removed a subscriber: gerritbot.
Jdforrester-WMF added a project: VisualEditor 2014/15 Q3 blockers.Mar 20 2015, 3:51 PM
Jdforrester-WMF edited a custom field.Mar 20 2015, 3:54 PM
Jdforrester-WMF moved this task from Nominated to Done on the VisualEditor 2014/15 Q3 blockers board.
Krinkle moved this task from Untriaged to Done on the Continuous-Integration-Infrastructure board.Mar 20 2015, 11:02 PM
Krinkle mentioned this in rCIJEafc1ef05ff14: Use EXECUTOR_NUMBER instead of BUILD_TAG in TMPDIR.Mar 24 2015, 5:39 PM
Jdforrester-WMF added a project: WMF-deploy-2015-03-25_(1.25wmf23).Mar 25 2015, 6:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits