Page MenuHomePhabricator

Create selenium-CirrusSearch-jessie daily Jenkins job
Closed, ResolvedPublic

Description

CirrusSearch uses Cucumber instead of Mocha.

As based on the parent task, we'll need new task to migrate browser tests from Jenkins that are making queries to beta structure, such as:

We have a concern that if we drop support for ruby and selenium, the above job will probably fail and we'll need to figure out what to to.

Details

Related Gerrit Patches:

Event Timeline

debt created this task.Sep 6 2017, 5:09 PM

Currently our tests that run on the beta cluster are managed by a common template for the ruby stack. This template detects the environment and runs a selected set of tests from the CirrusSearch test suite.
It's still unclear to me how this will work on top of the nodejs stack, there are documentations about setting up jenkins for javascript tests but I don't know if there are common practices for setting up multi environments config. @zeljkofilipin any suggestions?

The only Node.js+Selenium job targeting beta cluster is selenium-RelatedArticles-jessie (code). (The job is currently broken because of T176315, working on it.)

We do have support for multiple environments, at the moment mediawiki-vagrant (wdio.conf.js) and jenkins (wdio.conf.jenkins.js).

To run a subset of tests, a suite should be created in wdio.conf.js or wdio.conf.jenkins.js and mediawiki-extensions.yaml should be modified to run it.

Let me know as soon as you have any tests ready and we can work on making it happen.

Update: in addition to mediawiki-core-qunit-selenium-jessie Jenkins job, there is now also selenium-EXTENSION-jessie Jenkins job. I am also working on T179190: Run Selenium Cucumber tests in CI. Let me know if you have any questions.

β€’ zeljkofilipin renamed this task from Migrating browser tests from Jenkins on beta to Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch.Nov 14 2017, 12:23 PM
β€’ zeljkofilipin updated the task description. (Show Details)

Change 395872 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] [WIP] prepare replacement of selenium-CirrusSearch job

https://gerrit.wikimedia.org/r/395872

@zeljkofilipin we might be ready to port our selenium-CirrusSearch jenkins job to nodejs, I uploaded https://gerrit.wikimedia.org/r/#/c/395872/ to try to comply with the structure expected by jenkins:

  • tests in tests/selenium/specs
  • wdio config in tests/selenium/wdio.conf.jenkins.js

This is just a smoke test that targets the beta cluster.

This is a bit hackish (I created symlinks) because the whole suite is in tests/integration but this one is not yet ready to be run by jenkins. We'll need to create a docker image will all the necessary components to make this happen in jenkins:

  • mediawiki with multiple wikis
  • elasticsearch

@dcausse apologies for the slow reply. 🐌

I have created test job: selenium-CirrusSearch-jessie-395872. I will run the tests there.

Change 398030 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] WIP Create selenium-CirrusSearch-jessie Jenkins job

https://gerrit.wikimedia.org/r/398030

I have added readme files to tests/selenium and tests/integration with instructions on how to run tests targeting vagrant and beta cluster. I am not sure if the instructions there are correct. Please update the files.

Also, I am having trouble running the tests on my machine. I did upgrade to macOS 10.12.6 recently, and that did cause some trouble, so I am not sure if that is related or not.

~/Documents/gerrit/mediawiki/vagrant$ vagrant status
Current machine states:

default                   running (virtualbox)

The VM is running. To stop this VM, you can run `vagrant halt` to
shut it down forcefully, or you can run `vagrant suspend` to simply
suspend the virtual machine. In either case, to restart it again,
simply run `vagrant up`.

~/Documents/gerrit/mediawiki/vagrant$ vagrant roles list -e
Enabled roles:

cirrussearch    

~/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch$ node -v
v9.2.1

~/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch$ npm -v
5.6.0
~/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch$ npm i

> fibers@2.0.0 install /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/fibers
> node build.js || nodejs build.js

  CXX(target) Release/obj.target/fibers/src/fibers.o
../src/fibers.cc:318:3: warning: ignoring return value of function declared with 'warn_unused_result' attribute [-Wunused-result]
                object->SetAccessor(isolate->GetCurrentContext(), name, (AccessorNameGetterCallback)getter, (AccessorNameSetterCallba...
                ^~~~~~~~~~~~~~~~~~~
1 warning generated.
  CXX(target) Release/obj.target/fibers/src/coroutine.o
  CC(target) Release/obj.target/fibers/src/libcoro/coro.o
  SOLINK_MODULE(target) Release/fibers.node
Installed in `/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/fibers/bin/darwin-x64-59/fibers.node`

> dtrace-provider@0.8.5 install /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/dtrace-provider
> node scripts/install.js


> cucumber-expressions@5.0.7 postinstall /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/cucumber-expressions
> node scripts/postinstall.js


> cucumber-expressions@3.0.0 postinstall /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/wdio-cucumber-framework/node_modules/cucumber-expressions
> node scripts/postinstall.js

npm notice created a lockfile as package-lock.json. You should commit this file.
added 553 packages in 22.009s
~/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch$ npm run selenium

> cirrussearch@0.0.0 selenium /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch
> killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & grunt webdriver:test; killall chromedriver

Starting ChromeDriver 2.34.522932 (4140ab217e1ca1bec0c4b4d1b148f3361eb3a03e) on port 4444
Only local connections are allowed.
Running "webdriver:test" (webdriver) task

>> Something went wrong: listen EADDRINUSE /tmp/cirrussearch-integration-tagtracker
Warning: Task "webdriver:test" failed. Use --force to continue.

Aborted due to warnings.
sh: line 1:  9049 Terminated: 15          chromedriver --url-base=/wd/hub --port=4444

The error EADDRINUSE /tmp/cirrussearch-integration-tagtracker means that the tests are running in the background or that we failed to cleanup the socket when the tests finished or was killed.
It's perfectly fine to delete /tmp/cirrussearch-integration-tagtracker if you think the test is no longer running.

On my side I use :

vagrant@vagrantvirgo:/vagrant/mediawiki/extensions/CirrusSearch$ npm -v
2.15.12
vagrant@vagrantvirgo:/vagrant/mediawiki/extensions/CirrusSearch$ node -v
v6.12.0

Then for running tests I launch : grunt webdriver:test.
I'm not sure to understand why but running npm run selenium works but hitting Ctrl-C will keep the process running in the background for a while.

@dcausse I am still stuck. I do not know how to run tests on my machine.

cd mediawiki/vagrant
vagrant up
vagrant roles enable cirrussearch
vagrant provision
cd mediawiki/extensions/CirrusSearch
npm install

./node_modules/.bin/grunt webdriver:test
Running "webdriver:test" (webdriver) task

Server initialized
ERROR: connect ECONNREFUSED 127.0.0.1:4444
chrome
    at new RuntimeError (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/webdriverio/build/lib/utils/ErrorHandler.js:144:12)
    at Request._callback (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/webdriverio/build/lib/utils/RequestHandler.js:327:43)
    at self.callback (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/request/request.js:186:22)
    at Request.emit (events.js:159:13)
    at Request.onRequestError (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/request/request.js:878:8)
    at ClientRequest.emit (events.js:164:20)
    at Socket.socketErrorListener (_http_client.js:389:9)
    at Socket.emit (events.js:159:13)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at process._tickCallback (internal/process/next_tick.js:152:19)
...
$ CIRRUS_BETA_SMOKE_TEST=true ./node_modules/.bin/grunt webdriver:test
Running "webdriver:test" (webdriver) task
Server initialized
ERROR: connect ECONNREFUSED 127.0.0.1:4444
chrome
    at new RuntimeError (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/webdriverio/build/lib/utils/ErrorHandler.js:144:12)
    at Request._callback (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/webdriverio/build/lib/utils/RequestHandler.js:327:43)
    at self.callback (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/request/request.js:186:22)
    at Request.emit (events.js:159:13)
    at Request.onRequestError (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/request/request.js:878:8)
    at ClientRequest.emit (events.js:164:20)
    at Socket.socketErrorListener (_http_client.js:389:9)
    at Socket.emit (events.js:159:13)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at process._tickCallback (internal/process/next_tick.js:152:19)

Wrote xunit report to [../log/].
Warning: Task "webdriver:test" failed. Use --force to continue.

Aborted due to warnings.

chomedriver has to be started to listen on :4444. There might be some magic way to get nodejs to spawn this but I'm not sure how. I generally start it myself using:

chromedriver --port=4444 --url-base=/wd/hub

Argh. Of course. My mistake. Works fine now.

$ CIRRUS_BETA_SMOKE_TEST=true npm run selenium

> cirrussearch@0.0.0 selenium /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch
> killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & grunt webdriver:test; killall chromedriver

Starting ChromeDriver 2.34.522932 (4140ab217e1ca1bec0c4b4d1b148f3361eb3a03e) on port 4444
Only local connections are allowed.
Running "webdriver:test" (webdriver) task
Server initialized
GVA info: Successfully connected to the Intel plugin, offline Gen7 
------------------------------------------------------------------
[chrome #0-0] Session ID: ad06cf2e7d210716ba1fddb93129e08e
[chrome #0-0] Spec: /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/tests/integration/features/smoke.feature
[chrome #0-0] Running: chrome
[chrome #0-0]
[chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke: Smoke test
[chrome #0-0]
[chrome #0-0]     @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke, @en.wikipedia.beta.wmflabs.org: Search suggestions
[chrome #0-0]       βœ“ I am at a random page
[chrome #0-0]       βœ“ I type main p into the search box
[chrome #0-0]       βœ“ suggestions should appear
[chrome #0-0]       βœ“ Main Page is the first suggestion
[chrome #0-0]
[chrome #0-0]     @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke: Fill in search term and click search
[chrome #0-0]       βœ“ I am at a random page
[chrome #0-0]       βœ“ I type ma into the search box
[chrome #0-0]       βœ“ I click the search button
[chrome #0-0]       βœ“ I am on a page titled Search results
[chrome #0-0]
[chrome #0-0]     @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke, @en.wikipedia.beta.wmflabs.org: Search with accent yields result page with accent
[chrome #0-0]       βœ“ I am at a random page
[chrome #0-0]       βœ“ I search for África
[chrome #0-0]       βœ“ I am on a page titled África
[chrome #0-0]
[chrome #0-0]
[chrome #0-0] 11 passing (7s)
[chrome #0-0]

Wrote xunit report to [../log/].


11 passing (8.70s)


Done.
sh: line 1:  4642 Terminated: 15          chromedriver --url-base=/wd/hub --port=4444
β€’ zeljkofilipin added a comment.EditedDec 18 2017, 5:08 PM

The job is also green! πŸ’š selenium-CirrusSearch-jessie-395872

00:01:32.128 + CIRRUS_BETA_SMOKE_TEST=true
00:01:32.128 + npm run selenium
00:01:32.606 
00:01:32.606 > cirrussearch@0.0.0 selenium /home/jenkins/workspace/selenium-CirrusSearch-jessie-395872/src/extensions/CirrusSearch
00:01:32.606 > killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & grunt webdriver:test; killall chromedriver
00:01:32.606 
00:01:32.641 Starting ChromeDriver 2.27 (undefined) on port 4444
00:01:32.642 Only local connections are allowed.
00:01:33.393 Running "webdriver:test" (webdriver) task
00:01:34.225 Server initialized
00:01:43.511 ------------------------------------------------------------------
00:01:43.511 [chrome #0-0] Session ID: 3fb0d94b527cf231f637799a7cf4252d
00:01:43.512 [chrome #0-0] Spec: /home/jenkins/workspace/selenium-CirrusSearch-jessie-395872/src/extensions/CirrusSearch/tests/integration/features/smoke.feature
00:01:43.512 [chrome #0-0] Running: chrome
00:01:43.512 [chrome #0-0]
00:01:43.512 [chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke: Smoke test
00:01:43.512 [chrome #0-0]
00:01:43.512 [chrome #0-0]     @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke, @en.wikipedia.beta.wmflabs.org: Search suggestions
00:01:43.513 [chrome #0-0]       βœ“ I am at a random page
00:01:43.513 [chrome #0-0]       βœ“ I type main p into the search box
00:01:43.514 [chrome #0-0]       βœ“ suggestions should appear
00:01:43.514 [chrome #0-0]       βœ“ Main Page is the first suggestion
00:01:43.515 [chrome #0-0]
00:01:43.515 [chrome #0-0]     @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke: Fill in search term and click search
00:01:43.515 [chrome #0-0]       βœ“ I am at a random page
00:01:43.516 [chrome #0-0]       βœ“ I type ma into the search box
00:01:43.516 [chrome #0-0]       βœ“ I click the search button
00:01:43.517 [chrome #0-0]       βœ“ I am on a page titled Search results
00:01:43.517 [chrome #0-0]
00:01:43.517 [chrome #0-0]     @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke, @en.wikipedia.beta.wmflabs.org: Search with accent yields result page with accent
00:01:43.518 [chrome #0-0]       βœ“ I am at a random page
00:01:43.518 [chrome #0-0]       βœ“ I search for África
00:01:43.518 [chrome #0-0]       βœ“ I am on a page titled África
00:01:43.519 [chrome #0-0]
00:01:43.519 [chrome #0-0]
00:01:43.519 [chrome #0-0] 11 passing (8s)
00:01:43.520 [chrome #0-0]
00:01:43.520 
00:01:43.545 Wrote xunit report to [../log/].
00:01:43.545 
00:01:43.545 
00:01:43.546 11 passing (9.90s)
β€’ zeljkofilipin added a comment.EditedDec 20 2017, 3:09 PM

@dcausse @EBernhardson I am having trouble running CirrusSearch Selenium tests targeting mediawiki-vagrant. A lot of the tests are failing. Am I doing something wrong?

$ npm run selenium

> cirrussearch@0.0.0 selenium /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch
> killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & grunt webdriver:test; killall chromedriver

Starting ChromeDriver 2.34.522932 (4140ab217e1ca1bec0c4b4d1b148f3361eb3a03e) on port 4444
Only local connections are allowed.
Running "webdriver:test" (webdriver) task

Server initialized
GVA info: Successfully connected to the Intel plugin, offline Gen7 
[16:08:05] [S] [MWBOT] Login successful: Admin@http://cirrustest.wiki.local.wmftest.net:8080/w
[16:08:05] [?] [0001/0001] [DELETE]   DeleteMeRedirect (missingtitle)
[16:08:05] [D] The page you specified doesn't exist.
[16:08:05] [=] [0001/0001] [INCIRRUS] DeleteMeRedirect
Failed initializing tag @setup_main or @filters or @prefix or @bad_syntax or @wildcard or @exact_quotes or @phrase_prefix:  { Error: invalidjson: No valid JSON response
    at rawRequest.then (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/mwbot/src/index.js:249:31)
    at tryCatcher (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:773:18)
    at tryOnImmediate (timers.js:734:5)
    at processImmediate [as _immediateCallback] (timers.js:711:5)
  code: 'invalidjson',
  info: 'No valid JSON response',
  response: 'Class undefined: PoolCounter_Client\n' }
[16:08:06] [=] [0001/0018] [EDIT]     Template:Template Test
...

Change 399431 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Add ansicolor, timeout and timestamps to selenium-{name}-jessie

https://gerrit.wikimedia.org/r/399431

Change 399431 merged by jenkins-bot:
[integration/config@master] Add ansicolor, timeout and timestamps to selenium-{name}-jessie

https://gerrit.wikimedia.org/r/399431

Change 399656 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Remove ESLint inline comments in WebdriverIO configuration files

https://gerrit.wikimedia.org/r/399656

Change 399656 merged by jenkins-bot:
[mediawiki/core@master] Remove ESLint inline comments in WebdriverIO configuration files

https://gerrit.wikimedia.org/r/399656

Change 399833 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] WIP Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch

https://gerrit.wikimedia.org/r/399833

Change 395872 abandoned by DCausse:
Prepare replacement of selenium-CirrusSearch job

Reason:
as per discussion trying to reuse existing code base is not the best solution, writing core compatible tests is preferable. I'll take over Ide18493 and migrate the few tests needed to be run daily on beta.

https://gerrit.wikimedia.org/r/395872

β€’ zeljkofilipin renamed this task from Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch to Create selenium-CirrusSearch-jessie daily Jenkins job.Jan 16 2018, 10:55 AM

Change 381785 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/extensions/CirrusSearch@master] WIP Selenium smoke test

https://gerrit.wikimedia.org/r/381785

Change 399833 abandoned by Zfilipin:
WIP Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch

https://gerrit.wikimedia.org/r/399833

Change 398030 merged by jenkins-bot:
[integration/config@master] Create selenium-CirrusSearch-jessie Jenkins job

https://gerrit.wikimedia.org/r/398030

Change 381785 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Selenium smoke test

https://gerrit.wikimedia.org/r/381785

β€’ zeljkofilipin closed this task as Resolved.Jan 17 2018, 3:40 PM

The job is deployed and running: selenium-CirrusSearch-jessie. It fails because of T185055: Stack overflow when Redis is down.