migrate mwext-mw-selenium to Nodepool instances
Closed, ResolvedPublic

Description

Splitted from T54425, the mwext-mw-selenium jobs need to be migrated to Nodepool instances

Oct 26th, https://gerrit.wikimedia.org/r/#/c/293096/ adds new jobs in the experimental pipeline

Experimental builds:

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Zppix. · View Herald TranscriptJun 6 2016, 3:14 PM

Change 292927 had a related patch set uploaded (by Hashar):
(WIP) mwext-mw-selenium to Nodepool (WIP)

https://gerrit.wikimedia.org/r/292927

Paladox added a subscriber: Paladox.Jun 6 2016, 3:17 PM

Change 292927 merged by jenkins-bot:
mwext-mw-selenium to Nodepool

https://gerrit.wikimedia.org/r/292927

Change 292950 had a related patch set uploaded (by Hashar):
Revert "mwext-mw-selenium to Nodepool"

https://gerrit.wikimedia.org/r/292950

Change 292950 merged by Hashar:
Revert "mwext-mw-selenium to Nodepool"

https://gerrit.wikimedia.org/r/292950

gem2.0 command not found, that is due to the switch from Trusty to Jessie... Need more work.

hashar claimed this task.

Change 293096 had a related patch set uploaded (by Hashar):
mwext-mw-selenium to Nodepool

https://gerrit.wikimedia.org/r/293096

Eventually I had to start fresh regarding how we run bundler/cucumber

Tried on https://integration.wikimedia.org/ci/job/mwext-mw-selenium-jessie/7/console and that fails with:

avconv not found on your system. Install it or change video recorder provider (Headless::Exception)
vendor/bundle/gems/headless-2.2.3/lib/headless/cli_util.rb:9:in `ensure_application_exists!'
vendor/bundle/gems/headless-2.2.3/lib/headless/video/video_recorder.rb:27:in `initialize'
vendor/bundle/gems/headless-2.2.3/lib/headless.rb:141:in `new'
vendor/bundle/gems/headless-2.2.3/lib/headless.rb:141:in `video'
vendor/bundle/gems/mediawiki_selenium-1.7.1/lib/mediawiki_selenium/helpers/headless_helper.rb:112:in `teardown'
vendor/bundle/gems/mediawiki_selenium-1.7.1/lib/mediawiki_selenium/helpers/screenshot_helper.rb:37:in `teardown'
vendor/bundle/gems/mediawiki_selenium-1.7.1/lib/mediawiki_selenium/cucumber/hooks.rb:69:in `After'

That is used for Selenium recording ( T113520 ).
And currently we have:

productionffmpeg
permanent slaveslibav-tools
nodepoolnone

ffmpeg did not make it to Jessie though it is available in jessie-backports. I am wondering whether the gem headless can be pointed to ffmpeg.

Apparently with jessie-backports we can co install libav-tools and ffmpeg \o/

Change 293146 had a related patch set uploaded (by Hashar):
dib: libav-tools on Jessie for Selenium screenshots

https://gerrit.wikimedia.org/r/293146

Change 293146 merged by jenkins-bot:
dib: libav-tools on Jessie for Selenium screenshots

https://gerrit.wikimedia.org/r/293146

Will need to refresh the Jessie image to have libav-tools provided but labs has some troubles right now T137241

https://gerrit.wikimedia.org/r/293146 is merged providing libav-tools I had the Jessie image refreshed yesterday around 16:00 UTC.

apt-cache policy libav-tools
libav-tools:
  Installed: 6:11.6-1~deb8u1
  Candidate: 6:11.6-1~deb8u1
  Version table:
     7:3.0.2-2~bpo8+1 0
        100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
 *** 6:11.6-1~deb8u1 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
        500 http://security.debian.org/ jessie/updates/main amd64 Packages
        100 /var/lib/dpkg/status
$ avconv
avconv version 11.6-6:11.6-1~deb8u1, Copyright (c) 2000-2014 the Libav developers
  built on Mar  2 2016 23:00:02 with gcc 4.9.2 (Debian 4.9.2-10)
Use -h to get full help or, even better, run 'man avconv'
Hyper fast Audio and Video encoder
usage: avconv [options] [[infile options] -i infile]... {[outfile options] outfile}...

I have confirmed the mwext-mw-selenium-jessie job got past the missing avconv.

Is this really in progress? I thought you are no longer working on it @hashar.

hashar changed the task status from Open to Stalled.Sep 15 2016, 4:14 PM
hashar removed hashar as the assignee of this task.

mwext-mw-selenium is stuck to the permanent Trusty slaves for now. That is until we get more nodes on Nodepool and play with it on the disposable slaves.

hashar changed the task status from Stalled to Open.Oct 26 2016, 1:55 PM
hashar claimed this task.

I have rebased the patch, the jobs for Nodepool are added in parallel and solely in the experimental pipeline so we can play test them.

As for load increase on Nodepool:

https://grafana.wikimedia.org/dashboard/db/zuul-job?var-pipeline=All&var-job=mwext-mw-selenium*&var-status=All shows mwext-mw-selenium* jobs had roughly 80 builds per day:

And https://grafana.wikimedia.org/dashboard/db/continuous-integration shows we have roughly 1000 builds per day on Nodepool

Change 293096 merged by jenkins-bot:
mwext-mw-selenium jobs on Nodepool

https://gerrit.wikimedia.org/r/293096

hashar updated the task description. (Show Details)Oct 26 2016, 2:04 PM
hashar updated the task description. (Show Details)Oct 26 2016, 2:14 PM

I have updated the task description to add a table listing the status of all repos having the mwext-mw-selenium jobs for Nodepool. If they all pass just fine, we can fully migrate and delete the version that runs on permanent slaves.

hashar updated the task description. (Show Details)Oct 26 2016, 2:19 PM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)Oct 26 2016, 2:24 PM
hashar updated the task description. (Show Details)Oct 26 2016, 2:50 PM

Looks failures are due to mediawiki/skins/Vector not being included. It is cloned after MW is installed and hence it is not added in LocalSettings.php.

So the job does:

- prepare-mediawiki-zuul-project-no-vendor
- zuul-cloner:
    projects: mediawiki/skins/Vector
- mw-selenium-nodepool

Thus install.php is run before Vector is injected and thus is not enabled in LocalSettings.php.

Change 327177 had a related patch set uploaded (by Hashar):
Properly inject SKIN_DEPENDENCIES

https://gerrit.wikimedia.org/r/327177

Change 327177 merged by jenkins-bot:
Properly inject SKIN_DEPENDENCIES

https://gerrit.wikimedia.org/r/327177

Change 327210 had a related patch set uploaded (by Hashar):
mwext-mw-selenium now get Vector dep from Zuul

https://gerrit.wikimedia.org/r/327210

Change 327210 merged by jenkins-bot:
mwext-mw-selenium now get Vector dep from Zuul

https://gerrit.wikimedia.org/r/327210

The jobs should now have Vector injected as a dependency and clone it before MediaWiki installer is run. That should fix the issue on Nodepool.

hashar updated the task description. (Show Details)Dec 14 2016, 3:49 PM

Tested a few extensions and they pass. I am going to do the switch and drop the old jobs.

Change 327220 had a related patch set uploaded (by Hashar):
mwext-mw-selenium jobs to Nodepool

https://gerrit.wikimedia.org/r/327220

Change 327220 merged by jenkins-bot:
mwext-mw-selenium jobs to Nodepool

https://gerrit.wikimedia.org/r/327220

hashar closed this task as Resolved.Dec 14 2016, 4:06 PM
hashar reopened this task as Open.Jan 5 2017, 1:06 PM

Still have to migrate the mediawiki-selenium-integration job for mediawiki/selenium.git . There is a non voting job mediawiki-selenium-integration-jessie which is broken due to lack of gem2.0.

Change 330675 had a related patch set uploaded (by Hashar):
Let nodepool selenium macro to change dir

https://gerrit.wikimedia.org/r/330675

Change 330675 merged by jenkins-bot:
Let nodepool selenium macro to change dir

https://gerrit.wikimedia.org/r/330675

Change 330856 had a related patch set uploaded (by Hashar):
test: invoke rspec directly

https://gerrit.wikimedia.org/r/330856

The Nodepool version of mediawiki-selenium-integration pass with the two above patches :]

Is this resolved?

Change 330856 merged by jenkins-bot:
[mediawiki/selenium@master] test: invoke rspec directly

https://gerrit.wikimedia.org/r/330856

I went ahead and just +2ed the patch.

The last remaining issue is PhantomJS not being available Jessie :-/ I have https://gerrit.wikimedia.org/r/#/c/330864/ that auto magically download it but that is really terrible.

Just found out that PhantomJS 2.1.1 is available in jessie-backports since March 8th!!! https://packages.debian.org/jessie-backports/phantomjs

Change 344613 had a related patch set uploaded (by Hashar):
[operations/puppet@production] contint: install PhantomJS from backport

https://gerrit.wikimedia.org/r/344613

Mentioned in SAL (#wikimedia-releng) [2017-03-27T10:47:21Z] <hashar> Updating Nodepool Jessie image to include PhantomJS - T137112

Change 344613 merged by Muehlenhoff:
[operations/puppet@production] contint: install PhantomJS from backport

https://gerrit.wikimedia.org/r/344613

Mentioned in SAL (#wikimedia-releng) [2017-03-27T10:47:21Z] <hashar> Updating Nodepool Jessie image to include PhantomJS - T137112

PhantomJS is not included in that snapshot.

Mentioned in SAL (#wikimedia-releng) [2017-03-27T10:59:36Z] <hashar> Updating Nodepool Jessie image to include PhantomJS (take two) - T137112

So we now have PhantomJS 2.11 but the mediawiki_selenium suite fails with:

Scenario: Navigating to the wiki                              # features/basic_usage.feature:25
  When I navigate to the `wiki_url`                           # features/step_definitions/browser_steps.rb:19
  Then the wiki page should have loaded                       # features/step_definitions/browser_steps.rb:31
    unexpected response, code=404, content-type="text/plain"
    Error - Unable to load Atom 'find_element' from file ':/ghostdriver/./third_party/webdriver-atoms/find_element.js' (Selenium::WebDriver::Error::WebDriverError)
    ./features/step_definitions/browser_steps.rb:32:in `/^the wiki page should have loaded$/'
    features/basic_usage.feature:27:in `Then the wiki page should have loaded'

Something something is broken. Maybe related to Ghostdriver / Phantomjs 2 :(

Reproduced on my local machine. From the package README.Debian:

Ghostdriver is crippled due to removed source-less pre-built blobs:

  src/ghostdriver/third_party/webdriver-atoms/*

Therefore all PDF functionality is broken.

Which would explain the 404 on /ghostdriver/./third_party/webdriver-atoms/find_element.js ...

The Debian source is in https://anonscm.debian.org/git/collab-maint/phantomjs.git . The upstream branches has the imported source code, the HEAD being:

(tag: upstream/2.1.1+dfsg, origin/upstream, upstream) Imported Upstream version 2.1.1+dfsg (1 year, 1 month ago) <Dmitry Smirnov>

The PhantomJS code has been imported but stripped from anything that is not canonical source. That is when it becomes messy:

The Deb package should use the canonical source most probably or tweak the package to depends on selenium which might chip the webdriver atoms already.

So in short PhantomJS, has provided by Debian, does not support Webdriver.

The mediawiki_selenium integration test that fails are in features/basic_usage.feature. They check env variables are probably recognized and we can do very basic operations. There is no reason to bind that to phantomjs and we can just switch to chrome.

Change 344941 had a related patch set uploaded (by Hashar):
[mediawiki/selenium@master] Switch integration test to Chrome

https://gerrit.wikimedia.org/r/344941

Change 344941 merged by jenkins-bot:
[mediawiki/selenium@master] Switch integration test to Chrome

https://gerrit.wikimedia.org/r/344941

Change 344944 had a related patch set uploaded (by Hashar):
[integration/config@master] mediawiki_selenium: promote Nodepool integration job

https://gerrit.wikimedia.org/r/344944

Change 344944 merged by jenkins-bot:
[integration/config@master] mediawiki_selenium: promote Nodepool integration job

https://gerrit.wikimedia.org/r/344944

Change 344946 had a related patch set uploaded (by Hashar):
[integration/config@master] Clean unused mediawik_selenium JJB macro

https://gerrit.wikimedia.org/r/344946

hashar closed this task as Resolved.Mar 27 2017, 1:45 PM

The job that ran on Jessie finally pass. I have dropped the one that ran on Trusty and cleaned out JJB configuration.

Change 344946 merged by jenkins-bot:
[integration/config@master] Clean unused mediawik_selenium JJB macro

https://gerrit.wikimedia.org/r/344946