Migrate selenium-Wikibase-chrome selenium-WikibaseLexeme-chrome to Docker containers
Open, NormalPublic

Description

selenium-Wikibase-chrome selenium-WikibaseLexeme-chrome must be migrated to Docker containers. They run rake selenium.

I am not sure why they have the -chrome suffix though.

hashar created this task.Nov 23 2018, 1:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 23 2018, 1:50 PM

I think -chrome is because they use a local browser (Chrome) while the default is using a remote browser on Sauce Labs.

greg triaged this task as Normal priority.Nov 29 2018, 4:42 AM
hashar updated the task description. (Show Details)Jan 9 2019, 3:08 PM

The job use ruby gem mdediawiki_selenium. I am not even sure how they hardcode chrome.

Wikibase has:

tests/browser/ci.yml
BROWSER:
  - chrome

MEDIAWIKI_ENVIRONMENT:
  - beta
  - test

PLATFORM:
  - Linux

Done two years ago with 591294fb61bf543f7077d68c753299f14a47fd00 for T147401: Wikibase browsertests are flaky in Firefox on test.wikidata.org

Difference between the job templates {name}-selenium and {name}-selenium-chrome:

@ -1,5 +1,5 @@
 - job-template:
-    name: 'selenium-{project}'
+    name: 'selenium-{project}-chrome'
     defaults: selenium
 
     axes:
@@ -7,7 +7,7 @@
           type: label-expression
           name: label
           values:
-            - BrowserTests
+            - DebianJessie && contintLabsSlave
       - axis:
           name: BROWSER
           type: yaml
@@ -22,7 +22,7 @@
           filename: '{path_to_ci_yml}'
 
     # Label for the parent job
-    node: BrowserTests
+    node: DebianJessie && contintLabsSlave
 
     builders:
       - shell: |
@@ -52,7 +52,20 @@
             exit 1
           fi
 
+          # screenshots
+          export SCREENSHOT_FAILURES=true
+          export SCREENSHOT_FAILURES_PATH="$WORKSPACE/log"
+
+          # videos
+          export SKIP_TMPFS=1
+          export HEADLESS=true
+          export HEADLESS_DISPLAY=$((70 + EXECUTOR_NUMBER % 20))
+          export HEADLESS_DESTROY_AT_EXIT=true
+          export HEADLESS_CAPTURE_PATH="$WORKSPACE/log"
+
+          # do not run at Sauce Labs
+          unset SAUCE_ONDEMAND_USERNAME
+
           # run the tests
           bundle exec rake selenium

The selenium-WikibaseLexeme-chrome job breaks since it tries to include Wikibase but it is not cloned. There is probably a bug about it. The repository has some wdio tests, so probably the selenium tests are less helpful.

Given the job is not working at all, I think we should just delete it. Not sure whom to reach out to to approve that.

For Wikibase, we would need a container that has ruby/bundler and chromium. There is none right now.

Note that https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/ has a fair amount of failures for both beta and test environments :/

Given the job is not working at all, I think we should just delete it. Not sure whom to reach out to to approve that.

@Addshore @Ladsgroup you were pretty active recently with Selenium tests. Can you approve this, or do you know who can?

@Addshore @Ladsgroup you were pretty active recently with Selenium tests. Can you approve this, or do you know who can?

IIRC, we are trying to make it pass, @WMDE-leszek was working on it.

@hashar Please do not remove these jobs. Those are daily jobs of Wikibase(Lexeme) extensions which we want to keep. It is true though, those have been red for more than a while. We're working on fixing those failures, to allow us to gradually migrate away from ruby tests to node ones.
It is actually my personal goal for this quarter to get all these daily selenium jobs of Wikibase's green. It is simply embarrassing that we haven't solved those problems for so long.

Regarding the -chrome bit of jobs: the idea has indeed been to use local browser to run tests targetting beta/test wikis, instead of running this via saucelabs. This has stemmed from constant problems we have had with saucelabs and Wikibase browser tests (there is quite a few of these tests which does not make a problem any easier). Jobs were timing out randomly, it was dificult to debug failures, and everything has generally been very slow. Hence the idea to run tests on WMF Jenkins infrastructure, using local browser.
I am not sure off top of my head how the job configure browser to be chromium. I am sure @zeljkofilipin did some magic back then :)

Finally, regarding containers: is there any container that has ruby/bundler and some local browser that could be used for our jobs? I believe (or hope) chromium is not the only one that works. Firefox would also do. The only requirement from Wikibase side is to be able to run the browser in the container, not go to saucelabs etc.

@hashar Please do not remove these jobs. Those are daily jobs of Wikibase(Lexeme) extensions which we want to keep. It is true though, those have been red for more than a while. We're working on fixing those failures, to allow us to gradually migrate away from ruby tests to node ones.
It is actually my personal goal for this quarter to get all these daily selenium jobs of Wikibase's green. It is simply embarrassing that we haven't solved those problems for so long.

No worries, I was just wondering whether those jobs still serve a purpose. They do so yeah we will keep them!

Regarding the -chrome bit of jobs: the idea has indeed been to use local browser to run tests targetting beta/test wikis, instead of running this via saucelabs. This has stemmed from constant problems we have had with saucelabs and Wikibase browser tests (there is quite a few of these tests which does not make a problem any easier). Jobs were timing out randomly, it was dificult to debug failures, and everything has generally been very slow. Hence the idea to run tests on WMF Jenkins infrastructure, using local browser.
I am not sure off top of my head how the job configure browser to be chromium. I am sure @zeljkofilipin did some magic back then :)

I can't tell why they have a -chrome suffix :-( Apparently we had WikibaseLexeme to default to use Firefox which I have changed to use Chrome https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseLexeme/+/457407/3/tests/browser/environments.yml That was to migrate mediawiki-selenium jobs that run on a per patch basis (and thus require a fully working mediawiki) T203178

I clearly remember SauceLabs to be an hassle, mostly due to the latency between our and Saucelabs and the insane amount of queries being done. That pills up and slows it down. IIRC the main benefits were screenshots and video recording which mediawiki-selenium gem supports nowaday.

Finally, regarding containers: is there any container that has ruby/bundler and some local browser that could be used for our jobs? I believe (or hope) chromium is not the only one that works. Firefox would also do. The only requirement from Wikibase side is to be able to run the browser in the container, not go to saucelabs etc.

I was looking into it this morning and felt like we would have to craft yet another container with ruby/bundler/browsers etc. But for T203178: Migrate Selenium tests based on the ruby library mediawiki_selenium I crafted a container that reuses the environment for Quibble, it just add ruby/bundler on top of it. So we should be able to use docker-registry.wikimedia.org/releng/quibble-stretch-bundle . I will craft some experimental jobs for that.

Thanks!

Change 484507 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: rake-mediawiki-selenium

https://gerrit.wikimedia.org/r/484507

hashar claimed this task.Jan 15 2019, 4:15 PM

I crafted a very basic container and got something to run with Wikibase. I will craft experimental Jenkins jobs and progress from there :-]

I also note that other mediawiki-selenium jobs are still running on permanent slaves (with label BrowserTests). So I guess I will migrate them as well.

Addshore moved this task from incoming to in progress on the Wikidata board.Mon, Feb 11, 9:04 AM