Page MenuHomePhabricator

Create "npm-browser" docker image with npm, xvfb, chromium, and firefox installed
Closed, ResolvedPublic

Description

The current npm-test image just has npm and ruby/jsduck installed and can do all the basic things MediaWiki extensions need, but we also need a version with xvfb/chromium/firefox to be able to run qunit (and selenium I'm guessing) too.

As an example mediawiki/extensions/WikibaseJavaScriptApi uses karma / phantomjs

Details

Related Gerrit Patches:
integration/config : masterMigrate npm-browser-test to Debian Stretch
integration/config : masterdocker: npm container based on stretch
integration/config : masterSwitch ArticlePlaceholder to npm-browser-test
integration/config : masternpm test with Xvfb and PhantomJS
integration/config : masterMove npm for data-values back to Nodepool

Event Timeline

Legoktm created this task.Oct 31 2017, 3:24 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 31 2017, 3:24 AM
hashar updated the task description. (Show Details)Nov 1 2017, 1:33 PM

Change 388185 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Move npm for data-values back to Nodepool

https://gerrit.wikimedia.org/r/388185

Change 388185 merged by jenkins-bot:
[integration/config@master] Move npm for data-values back to Nodepool

https://gerrit.wikimedia.org/r/388185

hashar added a comment.Nov 8 2017, 8:58 AM

The npm container is based on Jessie. I thought of having npm-browser to extend it, however Chromium is no more updated in Jessie (T170032).
So probably we want npm to be switched to stretch first.

Another issue is that I am not quite sure how the Docker entrypoint can spawns xvfb before running npm. But surely that is doable.

Paladox added a subscriber: Paladox.Nov 9 2017, 9:56 PM

Upstream gerrit use npm, xvfb but have google-chrome-stable installed. But same thing. you need to do --privilege when running the image otherwise chrome will fail to start.

hashar added a comment.EditedNov 10 2017, 7:47 AM

We can probably get some ideas from https://github.com/mark-adams/docker-chromium-xvfb

Given the first line of code I see is:

RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -

Hmm. No?

Regardless the useful bit of code is: https://github.com/mark-adams/docker-chromium-xvfb/blob/master/images/base/xvfb-chromium It is intended to be used as a replacement for bin/chromium : it spawns an Xvfb automatically and runs Chromium without a sandbox which I guess is not compatible with Docker.

One problem though is that we would have to keep regenerating the image whenever a new Chromium package is released. But surely that can be automatized somehow.

One problem though is that we would have to keep regenerating the image whenever a new Chromium package is released. But surely that can be automatized somehow.

The automation is proposed as T167504: New tool to track package updates/status for hosts and images (debmonitor).

Change 393232 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] npm test with Xvfb and PhantomJS

https://gerrit.wikimedia.org/r/393232

Change 393233 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Switch ArticlePlaceholder to npm-browser-test

https://gerrit.wikimedia.org/r/393233

I created a basic Docker container which starts Xvfb on display port 94 and runs the npm tests. That seems to pass fine on a few of the repositories that were affected.

Change 393232 merged by jenkins-bot:
[integration/config@master] npm test with Xvfb and PhantomJS

https://gerrit.wikimedia.org/r/393232

Change 393233 merged by jenkins-bot:
[integration/config@master] Switch ArticlePlaceholder to npm-browser-test

https://gerrit.wikimedia.org/r/393233

Upstream gerrit use npm, xvfb but have google-chrome-stable installed. But same thing. you need to do --privilege when running the image otherwise chrome will fail to start.

No --privilege is not a solution. Attempting to run chromium yields:

	Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted

Since a Docker container is already sandboxed, we have to find a way to pass to chrome --no-sandbox.

For Karma we have CHROME_BIN=/usr/bin/chromium, I guess we want a wrapper that pass --no-sandbox.

From the shell script /usr/bin/chromium we just have to:

ENV CHROMIUM_FLAGS="--no-sandbox"

Change 394338 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: npm container based on stretch

https://gerrit.wikimedia.org/r/394338

Change 394340 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] npm-browser with stretch

https://gerrit.wikimedia.org/r/394340

hashar added a comment.EditedNov 30 2017, 5:45 PM

I have been using data-values/value-view as a playground area. I have slightly edited it to run all browsers ( https://gerrit.wikimedia.org/r/#/c/394352/ )

diff --git a/Gruntfile.js b/Gruntfile.js
index e2fd710..ebc35f3 100644
--- a/Gruntfile.js
+++ b/Gruntfile.js
@@ -91,12 +91,12 @@ module.exports = function ( grunt ) {
                                frameworks: [ 'qunit' ]
                        },
                        all: {
-                               browsers: [ 'PhantomJS' ]
+                               browsers: ['PhantomJS', 'Chrome', 'Firefox']
                        }
                }
        } );
 
-       grunt.registerTask( 'test', [ 'eslint', 'jsonlint', 'banana', 'qunit' ] );
+       grunt.registerTask( 'test', [ 'qunit' ] );
        grunt.registerTask( 'qunit', 'karma' );
        grunt.registerTask( 'default', 'test' );
 };
diff --git a/package.json b/package.json
index 4e68b0f..8a848b0 100644
--- a/package.json
+++ b/package.json
@@ -33,7 +33,9 @@
     "grunt-karma": "^2.0.0",
     "karma": "^1.7.1",
     "karma-cli": "^1.0.1",
-    "karma-phantomjs-launcher": "^1.0.4",
+    "karma-chrome-launcher": "2.2.0",
+    "karma-firefox-launcher": "1.0.1",
+    "karma-phantomjs-launcher": "*",
     "karma-qunit": "^1.2.1",
     "qunitjs": "^2.4.0",
     "sinon": "^4.0.1"

Using https://gerrit.wikimedia.org/r/394340 that seems to work:

PhantomJS 2.1.1 (Linux 0.0.0): Executed 333 of 333 SUCCESS (5.253 secs / 5.169 secs)
Chrome 62.0.3202 (Linux 0.0.0): Executed 333 of 333 SUCCESS (4.605 secs / 4.51 secs)
Firefox 52.0.0 (Linux 0.0.0): Executed 333 of 333 SUCCESS (5.657 secs / 5.498 secs)

Mentioned in SAL (#wikimedia-releng) [2017-11-30T21:05:55Z] <hashar> docker push wmfreleng/npm-stretch:v2017.11.30.21.03 && docker push wmfreleng/npm-stretch:latest && docker push wmfreleng/npm-test-stretch:v2017.11.30.21.03 && docker push wmfreleng/npm-test-stretch:latest | https://gerrit.wikimedia.org/r/#/c/394338/ | T179360

Change 394338 merged by jenkins-bot:
[integration/config@master] docker: npm container based on stretch

https://gerrit.wikimedia.org/r/394338

Mentioned in SAL (#wikimedia-releng) [2017-12-01T10:08:41Z] <hashar> docker push wmfreleng/npm-browser-test-stretch:v2017.11.30.21.30 && docker push wmfreleng/npm-browser-test-stretch:latest | https://gerrit.wikimedia.org/r/#/c/394340/ | T179360

Mentioned in SAL (#wikimedia-releng) [2017-12-01T11:55:33Z] <hashar> updating *npm-browser-node-6-docker jobs to use a new container based on Stretch with Chromium/Firefox | https://gerrit.wikimedia.org/r/#/c/394340/ | T179360

Change 394340 merged by jenkins-bot:
[integration/config@master] Migrate npm-browser-test to Debian Stretch

https://gerrit.wikimedia.org/r/394340

hashar closed this task as Resolved.Dec 1 2017, 12:06 PM
hashar claimed this task.

Done! The *npm-browser-node-6-docker jobs are now based on Stretch with PhantomJS 2.1.1, Firefox 52 and Chromium 62. They have a Xvfb server on display :94.

Left to figure out later:

  • magically update Chromium when a new version is pushed to stretch. That will be made easier once we migrate to docker-pkg which has logic for that.
  • add chromedriver so we can migrate selenium webdriver.io jobs to it (ex: portals/deploy https://gerrit.wikimedia.org/r/#/c/394539/ T180777)