Page MenuHomePhabricator
Create Task
Maniphest T137112

migrate mwext-mw-selenium to Nodepool instances
Closed, ResolvedPublic
Actions

Description

Splitted from T54425, the mwext-mw-selenium jobs need to be migrated to Nodepool instances

Oct 26th, https://gerrit.wikimedia.org/r/#/c/293096/ adds new jobs in the experimental pipeline

Experimental builds:

RepoChangeResult
ArticlePlaceholderhttps://gerrit.wikimedia.org/r/#/c/317201/console failed lack of browser tests apparently
Flowhttps://gerrit.wikimedia.org/r/#/c/317919/PASS console
WikibaseMediaInfohttps://gerrit.wikimedia.org/r/#/c/317815/FAIL (miss Vector) console
Wikibasehttps://gerrit.wikimedia.org/r/#/c/317974/FAIL console
-extension-selenium zuul template:-
Echohttps://gerrit.wikimedia.org/r/#/c/313221/PASS console
MobileFrontendhttps://gerrit.wikimedia.org/r/#/c/327087/PASS console
MultimediaViewerhttps://gerrit.wikimedia.org/r/#/c/317355/PASS console
QuickSurveyshttps://gerrit.wikimedia.org/r/#/c/315390/FAIL console
RevisionSliderhttps://gerrit.wikimedia.org/r/#/c/317827/FAIL console
Popupshttps://gerrit.wikimedia.org/r/#/c/326919/PASS console
RelatedArticleshttps://gerrit.wikimedia.org/r/#/c/327094/PASS console

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Zppix. · View Herald TranscriptJun 6 2016, 3:14 PM
gerritbot subscribed.Jun 6 2016, 3:14 PM

Change 292927 had a related patch set uploaded (by Hashar):
(WIP) mwext-mw-selenium to Nodepool (WIP)

https://gerrit.wikimedia.org/r/292927

gerritbot added a project: Patch-For-Review.Jun 6 2016, 3:14 PM
Paladox subscribed.Jun 6 2016, 3:17 PM
gerritbot added a comment.Jun 6 2016, 3:45 PM

Change 292927 merged by jenkins-bot:
mwext-mw-selenium to Nodepool

https://gerrit.wikimedia.org/r/292927

hashar added a comment.Jun 6 2016, 3:47 PM

Did a recheck:

RepoChange rechecked
Echohttps://gerrit.wikimedia.org/r/#/c/292602/
MobileFrontendhttps://gerrit.wikimedia.org/r/#/c/286022/
MultimediaViewerhttps://gerrit.wikimedia.org/r/#/c/292704/
Popupshttps://gerrit.wikimedia.org/r/#/c/288724/
QuickSurveyshttps://gerrit.wikimedia.org/r/#/c/291453/
Wikibasehttps://gerrit.wikimedia.org/r/#/c/291502/
gerritbot added a comment.Jun 6 2016, 3:58 PM

Change 292950 had a related patch set uploaded (by Hashar):
Revert "mwext-mw-selenium to Nodepool"

https://gerrit.wikimedia.org/r/292950

gerritbot added a comment.Jun 6 2016, 4:00 PM

Change 292950 merged by Hashar:
Revert "mwext-mw-selenium to Nodepool"

https://gerrit.wikimedia.org/r/292950

hashar added a comment.Jun 6 2016, 4:01 PM

gem2.0 command not found, that is due to the switch from Trusty to Jessie... Need more work.

hashar added a parent task: T55697: [EPIC] trigger browser tests from Gerrit (tracking).Jun 6 2016, 9:42 PM
hashar mentioned this in T55697: [EPIC] trigger browser tests from Gerrit (tracking).
hashar claimed this task.Jun 6 2016, 9:45 PM
hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.
gerritbot added a comment.Jun 7 2016, 12:46 PM

Change 293096 had a related patch set uploaded (by Hashar):
mwext-mw-selenium to Nodepool

https://gerrit.wikimedia.org/r/293096

hashar added a comment.Jun 7 2016, 3:16 PM

Eventually I had to start fresh regarding how we run bundler/cucumber

Tried on https://integration.wikimedia.org/ci/job/mwext-mw-selenium-jessie/7/console and that fails with:

avconv not found on your system. Install it or change video recorder provider (Headless::Exception)
vendor/bundle/gems/headless-2.2.3/lib/headless/cli_util.rb:9:in `ensure_application_exists!'
vendor/bundle/gems/headless-2.2.3/lib/headless/video/video_recorder.rb:27:in `initialize'
vendor/bundle/gems/headless-2.2.3/lib/headless.rb:141:in `new'
vendor/bundle/gems/headless-2.2.3/lib/headless.rb:141:in `video'
vendor/bundle/gems/mediawiki_selenium-1.7.1/lib/mediawiki_selenium/helpers/headless_helper.rb:112:in `teardown'
vendor/bundle/gems/mediawiki_selenium-1.7.1/lib/mediawiki_selenium/helpers/screenshot_helper.rb:37:in `teardown'
vendor/bundle/gems/mediawiki_selenium-1.7.1/lib/mediawiki_selenium/cucumber/hooks.rb:69:in `After'

That is used for Selenium recording ( T113520 ).
And currently we have:

productionffmpeg
permanent slaveslibav-tools
nodepoolnone

ffmpeg did not make it to Jessie though it is available in jessie-backports. I am wondering whether the gem headless can be pointed to ffmpeg.

hashar added a comment.Jun 7 2016, 3:29 PM

Apparently with jessie-backports we can co install libav-tools and ffmpeg \o/

gerritbot added a comment.Jun 7 2016, 6:53 PM

Change 293146 had a related patch set uploaded (by Hashar):
dib: libav-tools on Jessie for Selenium screenshots

https://gerrit.wikimedia.org/r/293146

gerritbot added a comment.Jun 7 2016, 7:43 PM

Change 293146 merged by jenkins-bot:
dib: libav-tools on Jessie for Selenium screenshots

https://gerrit.wikimedia.org/r/293146

hashar added a comment.Jun 7 2016, 8:02 PM

Will need to refresh the Jessie image to have libav-tools provided but labs has some troubles right now T137241

hashar added a comment.Jun 10 2016, 8:59 AM

https://gerrit.wikimedia.org/r/293146 is merged providing libav-tools I had the Jessie image refreshed yesterday around 16:00 UTC.

apt-cache policy libav-tools
libav-tools:
  Installed: 6:11.6-1~deb8u1
  Candidate: 6:11.6-1~deb8u1
  Version table:
     7:3.0.2-2~bpo8+1 0
        100 http://mirrors.wikimedia.org/debian/ jessie-backports/main amd64 Packages
 *** 6:11.6-1~deb8u1 0
        500 http://mirrors.wikimedia.org/debian/ jessie/main amd64 Packages
        500 http://security.debian.org/ jessie/updates/main amd64 Packages
        100 /var/lib/dpkg/status
$ avconv
avconv version 11.6-6:11.6-1~deb8u1, Copyright (c) 2000-2014 the Libav developers
  built on Mar  2 2016 23:00:02 with gcc 4.9.2 (Debian 4.9.2-10)
Use -h to get full help or, even better, run 'man avconv'
Hyper fast Audio and Video encoder
usage: avconv [options] [[infile options] -i infile]... {[outfile options] outfile}...
hashar added a comment.Jun 10 2016, 9:06 AM

I have confirmed the mwext-mw-selenium-jessie job got past the missing avconv.

hashar moved this task from Inbox to In Progress on the Browser-Tests-Infrastructure board.Jun 10 2016, 9:07 AM
zeljkofilipin added a project: User-zeljkofilipin.Jul 7 2016, 1:50 PM
zeljkofilipin moved this task from Backlog 🪒 to Q1 👔 on the User-zeljkofilipin board.
zeljkofilipin moved this task from Q1 👔 to Backlog 🪒 on the User-zeljkofilipin board.Jul 12 2016, 12:00 PM
zeljkofilipin added a comment.Sep 15 2016, 4:12 PM

Is this really in progress? I thought you are no longer working on it @hashar.

hashar changed the task status from Open to Stalled.Sep 15 2016, 4:14 PM
hashar removed hashar as the assignee of this task.

mwext-mw-selenium is stuck to the permanent Trusty slaves for now. That is until we get more nodes on Nodepool and play with it on the disposable slaves.

zeljkofilipin moved this task from In Progress to CI on the Browser-Tests-Infrastructure board.Sep 15 2016, 4:22 PM
zeljkofilipin moved this task from In-progress to Backlog on the Continuous-Integration-Scaling board.
zeljkofilipin moved this task from Backlog 🪒 to Watching 📺 on the User-zeljkofilipin board.
hashar changed the task status from Stalled to Open.Oct 26 2016, 1:55 PM
hashar claimed this task.

I have rebased the patch, the jobs for Nodepool are added in parallel and solely in the experimental pipeline so we can play test them.

As for load increase on Nodepool:

https://grafana.wikimedia.org/dashboard/db/zuul-job?var-pipeline=All&var-job=mwext-mw-selenium*&var-status=All shows mwext-mw-selenium* jobs had roughly 80 builds per day:

mwext-mw-selenium_builds_per_day.png (252×644 px, 32 KB)

And https://grafana.wikimedia.org/dashboard/db/continuous-integration shows we have roughly 1000 builds per day on Nodepool

nodepool_vs_permanent_builds_per_day.png (270×424 px, 32 KB)

hashar moved this task from CI to In Progress on the Browser-Tests-Infrastructure board.Oct 26 2016, 1:57 PM
hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.
gerritbot added a comment.Oct 26 2016, 1:57 PM

Change 293096 merged by jenkins-bot:
mwext-mw-selenium jobs on Nodepool

https://gerrit.wikimedia.org/r/293096

hashar updated the task description. (Show Details)Oct 26 2016, 2:04 PM
hashar updated the task description. (Show Details)Oct 26 2016, 2:14 PM

I have updated the task description to add a table listing the status of all repos having the mwext-mw-selenium jobs for Nodepool. If they all pass just fine, we can fully migrate and delete the version that runs on permanent slaves.

hashar updated the task description. (Show Details)Oct 26 2016, 2:19 PM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)Oct 26 2016, 2:24 PM
hashar updated the task description. (Show Details)Oct 26 2016, 2:50 PM
hashar added a comment.Oct 26 2016, 3:30 PM

Looks failures are due to mediawiki/skins/Vector not being included. It is cloned after MW is installed and hence it is not added in LocalSettings.php.

hashar added a comment.Dec 1 2016, 10:35 AM

So the job does:

- prepare-mediawiki-zuul-project-no-vendor
- zuul-cloner:
    projects: mediawiki/skins/Vector
- mw-selenium-nodepool

Thus install.php is run before Vector is injected and thus is not enabled in LocalSettings.php.

gerritbot added a comment.Dec 14 2016, 2:52 PM

Change 327177 had a related patch set uploaded (by Hashar):
Properly inject SKIN_DEPENDENCIES

https://gerrit.wikimedia.org/r/327177

gerritbot added a comment.Dec 14 2016, 2:55 PM

Change 327177 merged by jenkins-bot:
Properly inject SKIN_DEPENDENCIES

https://gerrit.wikimedia.org/r/327177

gerritbot added a comment.Dec 14 2016, 3:21 PM

Change 327210 had a related patch set uploaded (by Hashar):
mwext-mw-selenium now get Vector dep from Zuul

https://gerrit.wikimedia.org/r/327210

gerritbot added a comment.Dec 14 2016, 3:27 PM

Change 327210 merged by jenkins-bot:
mwext-mw-selenium now get Vector dep from Zuul

https://gerrit.wikimedia.org/r/327210

hashar added a comment.Dec 14 2016, 3:28 PM

The jobs should now have Vector injected as a dependency and clone it before MediaWiki installer is run. That should fix the issue on Nodepool.

hashar updated the task description. (Show Details)Dec 14 2016, 3:49 PM

Tested a few extensions and they pass. I am going to do the switch and drop the old jobs.

gerritbot added a comment.Dec 14 2016, 3:57 PM

Change 327220 had a related patch set uploaded (by Hashar):
mwext-mw-selenium jobs to Nodepool

https://gerrit.wikimedia.org/r/327220

gerritbot added a comment.Dec 14 2016, 4:05 PM

Change 327220 merged by jenkins-bot:
mwext-mw-selenium jobs to Nodepool

https://gerrit.wikimedia.org/r/327220

hashar closed this task as Resolved.Dec 14 2016, 4:06 PM
hashar reopened this task as Open.Jan 5 2017, 1:06 PM

Still have to migrate the mediawiki-selenium-integration job for mediawiki/selenium.git . There is a non voting job mediawiki-selenium-integration-jessie which is broken due to lack of gem2.0.

gerritbot added a comment.Jan 5 2017, 2:33 PM

Change 330675 had a related patch set uploaded (by Hashar):
Let nodepool selenium macro to change dir

https://gerrit.wikimedia.org/r/330675

gerritbot added a comment.Jan 5 2017, 2:36 PM

Change 330675 merged by jenkins-bot:
Let nodepool selenium macro to change dir

https://gerrit.wikimedia.org/r/330675

gerritbot added a comment.Jan 6 2017, 11:16 AM

Change 330856 had a related patch set uploaded (by Hashar):
test: invoke rspec directly

https://gerrit.wikimedia.org/r/330856

hashar added a comment.Jan 6 2017, 12:47 PM

The Nodepool version of mediawiki-selenium-integration pass with the two above patches :]

zeljkofilipin added a comment.Mar 22 2017, 12:01 PM

Is this resolved?

hashar added a comment.Mar 22 2017, 5:29 PM

Pending:

test: invoke rspec directly
https://gerrit.wikimedia.org/r/330856

gerritbot added a comment.Mar 24 2017, 1:49 PM

Change 330856 merged by jenkins-bot:
[mediawiki/selenium@master] test: invoke rspec directly

https://gerrit.wikimedia.org/r/330856

hashar added a comment.Mar 24 2017, 1:50 PM

I went ahead and just +2ed the patch.

The last remaining issue is PhantomJS not being available Jessie :-/ I have https://gerrit.wikimedia.org/r/#/c/330864/ that auto magically download it but that is really terrible.

hashar added a comment.Mar 24 2017, 1:53 PM

Just found out that PhantomJS 2.1.1 is available in jessie-backports since March 8th!!! https://packages.debian.org/jessie-backports/phantomjs

gerritbot added a comment.Mar 24 2017, 2:03 PM

Change 344613 had a related patch set uploaded (by Hashar):
[operations/puppet@production] contint: install PhantomJS from backport

https://gerrit.wikimedia.org/r/344613

Stashbot subscribed.Mar 27 2017, 10:47 AM

Mentioned in SAL (#wikimedia-releng) [2017-03-27T10:47:21Z] <hashar> Updating Nodepool Jessie image to include PhantomJS - T137112

gerritbot added a comment.Mar 27 2017, 10:58 AM

Change 344613 merged by Muehlenhoff:
[operations/puppet@production] contint: install PhantomJS from backport

https://gerrit.wikimedia.org/r/344613

hashar added a comment.Mar 27 2017, 10:58 AM
In T137112#3133406, @Stashbot wrote:

Mentioned in SAL (#wikimedia-releng) [2017-03-27T10:47:21Z] <hashar> Updating Nodepool Jessie image to include PhantomJS - T137112

PhantomJS is not included in that snapshot.

Stashbot added a comment.Mar 27 2017, 10:59 AM

Mentioned in SAL (#wikimedia-releng) [2017-03-27T10:59:36Z] <hashar> Updating Nodepool Jessie image to include PhantomJS (take two) - T137112

hashar added a comment.Mar 27 2017, 12:29 PM

So we now have PhantomJS 2.11 but the mediawiki_selenium suite fails with:

Scenario: Navigating to the wiki                              # features/basic_usage.feature:25
  When I navigate to the `wiki_url`                           # features/step_definitions/browser_steps.rb:19
  Then the wiki page should have loaded                       # features/step_definitions/browser_steps.rb:31
    unexpected response, code=404, content-type="text/plain"
    Error - Unable to load Atom 'find_element' from file ':/ghostdriver/./third_party/webdriver-atoms/find_element.js' (Selenium::WebDriver::Error::WebDriverError)
    ./features/step_definitions/browser_steps.rb:32:in `/^the wiki page should have loaded$/'
    features/basic_usage.feature:27:in `Then the wiki page should have loaded'

Something something is broken. Maybe related to Ghostdriver / Phantomjs 2 :(

hashar added a comment.Mar 27 2017, 12:35 PM

Reproduced on my local machine. From the package README.Debian:

Ghostdriver is crippled due to removed source-less pre-built blobs:

  src/ghostdriver/third_party/webdriver-atoms/*

Therefore all PDF functionality is broken.

Which would explain the 404 on /ghostdriver/./third_party/webdriver-atoms/find_element.js ...

hashar added a comment.Mar 27 2017, 1:07 PM

The Debian source is in https://anonscm.debian.org/git/collab-maint/phantomjs.git . The upstream branches has the imported source code, the HEAD being:

(tag: upstream/2.1.1+dfsg, origin/upstream, upstream) Imported Upstream version 2.1.1+dfsg (1 year, 1 month ago) <Dmitry Smirnov>

The PhantomJS code has been imported but stripped from anything that is not canonical source. That is when it becomes messy:

The Deb package should use the canonical source most probably or tweak the package to depends on selenium which might chip the webdriver atoms already.

hashar added a comment.Mar 27 2017, 1:20 PM

So in short PhantomJS, has provided by Debian, does not support Webdriver.

The mediawiki_selenium integration test that fails are in features/basic_usage.feature. They check env variables are probably recognized and we can do very basic operations. There is no reason to bind that to phantomjs and we can just switch to chrome.

gerritbot added a comment.Mar 27 2017, 1:22 PM

Change 344941 had a related patch set uploaded (by Hashar):
[mediawiki/selenium@master] Switch integration test to Chrome

https://gerrit.wikimedia.org/r/344941

gerritbot added a comment.Mar 27 2017, 1:27 PM

Change 344941 merged by jenkins-bot:
[mediawiki/selenium@master] Switch integration test to Chrome

https://gerrit.wikimedia.org/r/344941

gerritbot added a comment.Mar 27 2017, 1:28 PM

Change 344944 had a related patch set uploaded (by Hashar):
[integration/config@master] mediawiki_selenium: promote Nodepool integration job

https://gerrit.wikimedia.org/r/344944

gerritbot added a comment.Mar 27 2017, 1:37 PM

Change 344944 merged by jenkins-bot:
[integration/config@master] mediawiki_selenium: promote Nodepool integration job

https://gerrit.wikimedia.org/r/344944

gerritbot added a comment.Mar 27 2017, 1:42 PM

Change 344946 had a related patch set uploaded (by Hashar):
[integration/config@master] Clean unused mediawik_selenium JJB macro

https://gerrit.wikimedia.org/r/344946

hashar closed this task as Resolved.Mar 27 2017, 1:45 PM

The job that ran on Jessie finally pass. I have dropped the one that ran on Trusty and cleaned out JJB configuration.

gerritbot added a comment.Mar 27 2017, 1:47 PM

Change 344946 merged by jenkins-bot:
[integration/config@master] Clean unused mediawik_selenium JJB macro

https://gerrit.wikimedia.org/r/344946

hashar mentioned this in T179465: Installing npm package phantomjs-prebuilt failing for WikibaseJavaScriptApi builds.Nov 1 2017, 10:37 AM
• Phabricator_maintenance added a project: RelEng-Archive-FY201718-Q2.Dec 21 2017, 6:57 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL