Page MenuHomePhabricator

Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x
Closed, ResolvedPublic

Description

Running tests on master branch on my machine:

P2395

$ source selenium_exports_for_vagrant.sh
$ bundle exec cucumber
...
Failing Scenarios:
cucumber features/commonterms_api.feature:24 # Scenario: The aggressive recall profile will display results even if some words are missing
cucumber features/did_you_mean_api.feature:100 # Scenario: When I use the laplace smoothing model with alpha value 0.000001 the suggestions are not complete
cucumber features/go_browser.feature:9 # Scenario: When I near match more than one page but one is exact (case, modulo case, or converted to title case) I go to that page
cucumber features/go_browser.feature:9 # Scenario: When I near match more than one page but one is exact (case, modulo case, or converted to title case) I go to that page
cucumber features/prefer_recent_api.feature:18 # Scenario: You can specify prefer-recent: in such a way that being super recent isn't enough
cucumber features/smoke.feature:23 # Scenario: Fill in search term and click search

579 scenarios (6 failed, 573 passed)
1842 steps (6 failed, 1836 passed)
10m44.793s

Running tests for commit 257928 on my machine:

P2397

$ bundle exec cucumber
...
Failing Scenarios:
cucumber features/commons.feature:7 # Scenario: A file that exists on commons and the local wiki returns the local result
cucumber features/commonterms_api.feature:24 # Scenario: The aggressive recall profile will display results even if some words are missing
cucumber features/full_text_browser.feature:70 # Scenario: Portions of file names
cucumber features/prefer_recent_api.feature:18 # Scenario: You can specify prefer-recent: in such a way that being super recent isn't enough
cucumber features/prefix_search_api.feature:3 # Scenario: Suggestions don't appear when you search for a string that is too long

577 scenarios (5 failed, 572 passed)
1834 steps (5 failed, 4 skipped, 1825 passed)
15m55.416s

I have run the tests three times in a row, got 5-8 failed scenarios, but none of the failing scenarios failed in all 3 test runs. :/

Output:

Event Timeline

dduvall raised the priority of this task from to Medium.
dduvall updated the task description. (Show Details)
zeljkofilipin moved this task from Inbox to Next on the Browser-Tests-Infrastructure board.
zeljkofilipin moved this task from Next to In Progress on the Browser-Tests-Infrastructure board.
zeljkofilipin set Security to None.
dduvall moved this task from In Progress to Next on the Browser-Tests-Infrastructure board.

@dduvall: I was sure you had a WIP commit in Gerrit related to this, but I can not find it.

According to the single job we have https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/ , there is only a couple of smoke tests running under Jenkins. Should not be too much of an hassle :-}

Change 257928 had a related patch set uploaded (by Zfilipin):
WIP Upgrade browser tests to the latest version of mediawiki_selenium

https://gerrit.wikimedia.org/r/257928

Change 257928 merged by jenkins-bot:
Upgrade browser tests to the latest version of mediawiki_selenium

https://gerrit.wikimedia.org/r/257928