CirrusSearch uses Cucumber instead of Mocha.
As based on the parent task, we'll need new task to migrate browser tests from Jenkins that are making queries to beta structure, such as:
We have a concern that if we drop support for ruby and selenium, the above job will probably fail and we'll need to figure out what to to.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | zeljkofilipin | T182986 Selenium framework improvements | |||
| Resolved | zeljkofilipin | T190994 Q4 Selenium framework improvements | |||
| Resolved | zeljkofilipin | T164721 Run Selenium tests in CI per-patch for extensions | |||
| Resolved | zeljkofilipin | T175179 Create selenium-CirrusSearch-jessie daily Jenkins job |
Currently our tests that run on the beta cluster are managed by a common template for the ruby stack. This template detects the environment and runs a selected set of tests from the CirrusSearch test suite.
It's still unclear to me how this will work on top of the nodejs stack, there are documentations about setting up jenkins for javascript tests but I don't know if there are common practices for setting up multi environments config. @zeljkofilipin any suggestions?
The only Node.js+Selenium job targeting beta cluster is selenium-RelatedArticles-jessie (code). (The job is currently broken because of T176315, working on it.)
We do have support for multiple environments, at the moment mediawiki-vagrant (wdio.conf.js) and jenkins (wdio.conf.jenkins.js).
To run a subset of tests, a suite should be created in wdio.conf.js or wdio.conf.jenkins.js and mediawiki-extensions.yaml should be modified to run it.
Update: in addition to mediawiki-core-qunit-selenium-jessie Jenkins job, there is now also selenium-EXTENSION-jessie Jenkins job. I am also working on T179190: Run Selenium Cucumber tests in CI. Let me know if you have any questions.
Change 395872 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] [WIP] prepare replacement of selenium-CirrusSearch job
@zeljkofilipin we might be ready to port our selenium-CirrusSearch jenkins job to nodejs, I uploaded https://gerrit.wikimedia.org/r/#/c/395872/ to try to comply with the structure expected by jenkins:
This is just a smoke test that targets the beta cluster.
This is a bit hackish (I created symlinks) because the whole suite is in tests/integration but this one is not yet ready to be run by jenkins. We'll need to create a docker image will all the necessary components to make this happen in jenkins:
@dcausse apologies for the slow reply. 🐌
I have created test job: selenium-CirrusSearch-jessie-395872. I will run the tests there.
Change 398030 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] WIP Create selenium-CirrusSearch-jessie Jenkins job
I have added readme files to tests/selenium and tests/integration with instructions on how to run tests targeting vagrant and beta cluster. I am not sure if the instructions there are correct. Please update the files.
Also, I am having trouble running the tests on my machine. I did upgrade to macOS 10.12.6 recently, and that did cause some trouble, so I am not sure if that is related or not.
~/Documents/gerrit/mediawiki/vagrant$ vagrant status Current machine states: default running (virtualbox) The VM is running. To stop this VM, you can run `vagrant halt` to shut it down forcefully, or you can run `vagrant suspend` to simply suspend the virtual machine. In either case, to restart it again, simply run `vagrant up`. ~/Documents/gerrit/mediawiki/vagrant$ vagrant roles list -e Enabled roles: cirrussearch ~/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch$ node -v v9.2.1 ~/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch$ npm -v 5.6.0
~/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch$ npm i
> fibers@2.0.0 install /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/fibers
> node build.js || nodejs build.js
CXX(target) Release/obj.target/fibers/src/fibers.o
../src/fibers.cc:318:3: warning: ignoring return value of function declared with 'warn_unused_result' attribute [-Wunused-result]
object->SetAccessor(isolate->GetCurrentContext(), name, (AccessorNameGetterCallback)getter, (AccessorNameSetterCallba...
^~~~~~~~~~~~~~~~~~~
1 warning generated.
CXX(target) Release/obj.target/fibers/src/coroutine.o
CC(target) Release/obj.target/fibers/src/libcoro/coro.o
SOLINK_MODULE(target) Release/fibers.node
Installed in `/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/fibers/bin/darwin-x64-59/fibers.node`
> dtrace-provider@0.8.5 install /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/dtrace-provider
> node scripts/install.js
> cucumber-expressions@5.0.7 postinstall /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/cucumber-expressions
> node scripts/postinstall.js
> cucumber-expressions@3.0.0 postinstall /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/wdio-cucumber-framework/node_modules/cucumber-expressions
> node scripts/postinstall.js
npm notice created a lockfile as package-lock.json. You should commit this file.
added 553 packages in 22.009s~/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch$ npm run selenium > cirrussearch@0.0.0 selenium /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch > killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & grunt webdriver:test; killall chromedriver Starting ChromeDriver 2.34.522932 (4140ab217e1ca1bec0c4b4d1b148f3361eb3a03e) on port 4444 Only local connections are allowed. Running "webdriver:test" (webdriver) task >> Something went wrong: listen EADDRINUSE /tmp/cirrussearch-integration-tagtracker Warning: Task "webdriver:test" failed. Use --force to continue. Aborted due to warnings. sh: line 1: 9049 Terminated: 15 chromedriver --url-base=/wd/hub --port=4444
The error EADDRINUSE /tmp/cirrussearch-integration-tagtracker means that the tests are running in the background or that we failed to cleanup the socket when the tests finished or was killed.
It's perfectly fine to delete /tmp/cirrussearch-integration-tagtracker if you think the test is no longer running.
On my side I use :
vagrant@vagrantvirgo:/vagrant/mediawiki/extensions/CirrusSearch$ npm -v 2.15.12 vagrant@vagrantvirgo:/vagrant/mediawiki/extensions/CirrusSearch$ node -v v6.12.0
Then for running tests I launch : grunt webdriver:test.
I'm not sure to understand why but running npm run selenium works but hitting Ctrl-C will keep the process running in the background for a while.
@dcausse I am still stuck. I do not know how to run tests on my machine.
cd mediawiki/vagrant
vagrant up
vagrant roles enable cirrussearch
vagrant provision
cd mediawiki/extensions/CirrusSearch
npm install
./node_modules/.bin/grunt webdriver:test
Running "webdriver:test" (webdriver) task
Server initialized
ERROR: connect ECONNREFUSED 127.0.0.1:4444
chrome
at new RuntimeError (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/webdriverio/build/lib/utils/ErrorHandler.js:144:12)
at Request._callback (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/webdriverio/build/lib/utils/RequestHandler.js:327:43)
at self.callback (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/request/request.js:186:22)
at Request.emit (events.js:159:13)
at Request.onRequestError (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/request/request.js:878:8)
at ClientRequest.emit (events.js:164:20)
at Socket.socketErrorListener (_http_client.js:389:9)
at Socket.emit (events.js:159:13)
at emitErrorNT (internal/streams/destroy.js:64:8)
at process._tickCallback (internal/process/next_tick.js:152:19)
...$ CIRRUS_BETA_SMOKE_TEST=true ./node_modules/.bin/grunt webdriver:test Running "webdriver:test" (webdriver) task Server initialized ERROR: connect ECONNREFUSED 127.0.0.1:4444 chrome at new RuntimeError (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/webdriverio/build/lib/utils/ErrorHandler.js:144:12) at Request._callback (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/webdriverio/build/lib/utils/RequestHandler.js:327:43) at self.callback (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/request/request.js:186:22) at Request.emit (events.js:159:13) at Request.onRequestError (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/request/request.js:878:8) at ClientRequest.emit (events.js:164:20) at Socket.socketErrorListener (_http_client.js:389:9) at Socket.emit (events.js:159:13) at emitErrorNT (internal/streams/destroy.js:64:8) at process._tickCallback (internal/process/next_tick.js:152:19) Wrote xunit report to [../log/]. Warning: Task "webdriver:test" failed. Use --force to continue. Aborted due to warnings.
chomedriver has to be started to listen on :4444. There might be some magic way to get nodejs to spawn this but I'm not sure how. I generally start it myself using:
chromedriver --port=4444 --url-base=/wd/hub
Argh. Of course. My mistake. Works fine now.
$ CIRRUS_BETA_SMOKE_TEST=true npm run selenium > cirrussearch@0.0.0 selenium /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch > killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & grunt webdriver:test; killall chromedriver Starting ChromeDriver 2.34.522932 (4140ab217e1ca1bec0c4b4d1b148f3361eb3a03e) on port 4444 Only local connections are allowed. Running "webdriver:test" (webdriver) task Server initialized GVA info: Successfully connected to the Intel plugin, offline Gen7 ------------------------------------------------------------------ [chrome #0-0] Session ID: ad06cf2e7d210716ba1fddb93129e08e [chrome #0-0] Spec: /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/tests/integration/features/smoke.feature [chrome #0-0] Running: chrome [chrome #0-0] [chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke: Smoke test [chrome #0-0] [chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke, @en.wikipedia.beta.wmflabs.org: Search suggestions [chrome #0-0] ✓ I am at a random page [chrome #0-0] ✓ I type main p into the search box [chrome #0-0] ✓ suggestions should appear [chrome #0-0] ✓ Main Page is the first suggestion [chrome #0-0] [chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke: Fill in search term and click search [chrome #0-0] ✓ I am at a random page [chrome #0-0] ✓ I type ma into the search box [chrome #0-0] ✓ I click the search button [chrome #0-0] ✓ I am on a page titled Search results [chrome #0-0] [chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke, @en.wikipedia.beta.wmflabs.org: Search with accent yields result page with accent [chrome #0-0] ✓ I am at a random page [chrome #0-0] ✓ I search for África [chrome #0-0] ✓ I am on a page titled África [chrome #0-0] [chrome #0-0] [chrome #0-0] 11 passing (7s) [chrome #0-0] Wrote xunit report to [../log/]. 11 passing (8.70s) Done. sh: line 1: 4642 Terminated: 15 chromedriver --url-base=/wd/hub --port=4444
The job is also green! 💚 selenium-CirrusSearch-jessie-395872
00:01:32.128 + CIRRUS_BETA_SMOKE_TEST=true 00:01:32.128 + npm run selenium 00:01:32.606 00:01:32.606 > cirrussearch@0.0.0 selenium /home/jenkins/workspace/selenium-CirrusSearch-jessie-395872/src/extensions/CirrusSearch 00:01:32.606 > killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & grunt webdriver:test; killall chromedriver 00:01:32.606 00:01:32.641 Starting ChromeDriver 2.27 (undefined) on port 4444 00:01:32.642 Only local connections are allowed. 00:01:33.393 Running "webdriver:test" (webdriver) task 00:01:34.225 Server initialized 00:01:43.511 ------------------------------------------------------------------ 00:01:43.511 [chrome #0-0] Session ID: 3fb0d94b527cf231f637799a7cf4252d 00:01:43.512 [chrome #0-0] Spec: /home/jenkins/workspace/selenium-CirrusSearch-jessie-395872/src/extensions/CirrusSearch/tests/integration/features/smoke.feature 00:01:43.512 [chrome #0-0] Running: chrome 00:01:43.512 [chrome #0-0] 00:01:43.512 [chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke: Smoke test 00:01:43.512 [chrome #0-0] 00:01:43.512 [chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke, @en.wikipedia.beta.wmflabs.org: Search suggestions 00:01:43.513 [chrome #0-0] ✓ I am at a random page 00:01:43.513 [chrome #0-0] ✓ I type main p into the search box 00:01:43.514 [chrome #0-0] ✓ suggestions should appear 00:01:43.514 [chrome #0-0] ✓ Main Page is the first suggestion 00:01:43.515 [chrome #0-0] 00:01:43.515 [chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke: Fill in search term and click search 00:01:43.515 [chrome #0-0] ✓ I am at a random page 00:01:43.516 [chrome #0-0] ✓ I type ma into the search box 00:01:43.516 [chrome #0-0] ✓ I click the search button 00:01:43.517 [chrome #0-0] ✓ I am on a page titled Search results 00:01:43.517 [chrome #0-0] 00:01:43.517 [chrome #0-0] @clean, @firefox, @test2.wikipedia.org, @phantomjs, @smoke, @en.wikipedia.beta.wmflabs.org: Search with accent yields result page with accent 00:01:43.518 [chrome #0-0] ✓ I am at a random page 00:01:43.518 [chrome #0-0] ✓ I search for África 00:01:43.518 [chrome #0-0] ✓ I am on a page titled África 00:01:43.519 [chrome #0-0] 00:01:43.519 [chrome #0-0] 00:01:43.519 [chrome #0-0] 11 passing (8s) 00:01:43.520 [chrome #0-0] 00:01:43.520 00:01:43.545 Wrote xunit report to [../log/]. 00:01:43.545 00:01:43.545 00:01:43.546 11 passing (9.90s)
@dcausse @EBernhardson I am having trouble running CirrusSearch Selenium tests targeting mediawiki-vagrant. A lot of the tests are failing. Am I doing something wrong?
$ npm run selenium > cirrussearch@0.0.0 selenium /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch > killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & grunt webdriver:test; killall chromedriver Starting ChromeDriver 2.34.522932 (4140ab217e1ca1bec0c4b4d1b148f3361eb3a03e) on port 4444 Only local connections are allowed. Running "webdriver:test" (webdriver) task Server initialized GVA info: Successfully connected to the Intel plugin, offline Gen7 [16:08:05] [S] [MWBOT] Login successful: Admin@http://cirrustest.wiki.local.wmftest.net:8080/w [16:08:05] [?] [0001/0001] [DELETE] DeleteMeRedirect (missingtitle) [16:08:05] [D] The page you specified doesn't exist. [16:08:05] [=] [0001/0001] [INCIRRUS] DeleteMeRedirect Failed initializing tag @setup_main or @filters or @prefix or @bad_syntax or @wildcard or @exact_quotes or @phrase_prefix: { Error: invalidjson: No valid JSON response at rawRequest.then (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/mwbot/src/index.js:249:31) at tryCatcher (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/util.js:16:23) at Promise._settlePromiseFromHandler (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/promise.js:512:31) at Promise._settlePromise (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/promise.js:569:18) at Promise._settlePromise0 (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/promise.js:614:10) at Promise._settlePromises (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/promise.js:693:18) at Async._drainQueue (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/async.js:133:16) at Async._drainQueues (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/async.js:143:10) at Immediate.Async.drainQueues [as _onImmediate] (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/node_modules/bluebird/js/release/async.js:17:14) at runCallback (timers.js:773:18) at tryOnImmediate (timers.js:734:5) at processImmediate [as _immediateCallback] (timers.js:711:5) code: 'invalidjson', info: 'No valid JSON response', response: 'Class undefined: PoolCounter_Client\n' } [16:08:06] [=] [0001/0018] [EDIT] Template:Template Test ...
Change 399431 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Add ansicolor, timeout and timestamps to selenium-{name}-jessie
Change 399431 merged by jenkins-bot:
[integration/config@master] Add ansicolor, timeout and timestamps to selenium-{name}-jessie
Change 399656 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Remove ESLint inline comments in WebdriverIO configuration files
Change 399656 merged by jenkins-bot:
[mediawiki/core@master] Remove ESLint inline comments in WebdriverIO configuration files
Change 399833 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] WIP Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch
Change 395872 abandoned by DCausse:
Prepare replacement of selenium-CirrusSearch job
Reason:
as per discussion trying to reuse existing code base is not the best solution, writing core compatible tests is preferable. I'll take over Ide18493 and migrate the few tests needed to be run daily on beta.
Change 381785 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/extensions/CirrusSearch@master] WIP Selenium smoke test
Change 399833 abandoned by Zfilipin:
WIP Run selenium-EXTENSION-jessie Jenkins job for CirrusSearch
Change 398030 merged by jenkins-bot:
[integration/config@master] Create selenium-CirrusSearch-jessie Jenkins job
Change 381785 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Selenium smoke test
The job is deployed and running: selenium-CirrusSearch-jessie. It fails because of T185055: Stack overflow when Redis is down.
Looks like the problem is resolved, the job is green! 💚
https://integration.wikimedia.org/ci/view/Selenium/job/selenium-CirrusSearch-jessie/