Page MenuHomePhabricator

Prepare selenium browser test for page issues A/B test
Closed, ResolvedPublic3 Estimated Story Points

Description

The tests that fail vary but 3 out of the last 5 failures have been page issue specific.

https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue/624/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/consoleFull

15:21:35 @integration @chrome @en.m.wikipedia.beta.wmflabs.org @firefox @test2.m.wikipedia.org @vagrant
15:21:35 Feature: Issues
15:21:35 
15:21:35   Background:                                      # tests/browser/features/issues.feature:4
15:21:52     Given I am using the mobile site               # tests/browser/features/step_definitions/common_steps.rb:40
15:21:52     And I am on a page which has cleanup templates # tests/browser/features/step_definitions/create_page_api_steps.rb:30
15:21:52     And this page has issues                       # tests/browser/features/step_definitions/issues_steps.rb:13
15:21:52       timed out after 5 seconds, waiting for {:css=>".mw-mf-cleanup"} to become present (Watir::Wait::TimeoutError)
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/watir-webdriver-0.9.1/lib/watir-webdriver/wait.rb:44:in `until'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/watir-webdriver-0.9.1/lib/watir-webdriver/wait.rb:223:in `wait_until_present'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/page-object-1.1.0/lib/page-object/platforms/watir_webdriver/element.rb:160:in `when_present'
15:21:52       /srv/jenkins-workspace/workspace/selenium-MinervaNeue/BROWSER/chrome/MEDIAWIKI_ENVIRONMENT/beta/PLATFORM/Linux/label/BrowserTests/tests/browser/features/step_definitions/issues_steps.rb:14:in `block in <top (required)>'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `instance_exec'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `block in cucumber_instance_exec'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:69:in `cucumber_run_with_backtrace_filtering'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:36:in `cucumber_instance_exec'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/rb_support/rb_step_definition.rb:97:in `invoke'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/step_match.rb:25:in `invoke'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:60:in `invoke'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:38:in `accept'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:106:in `block in visit_step'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:105:in `visit_step'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:19:in `block in accept'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `each'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `accept'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:100:in `block in visit_steps'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:99:in `visit_steps'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:44:in `block in accept'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:54:in `block in with_visitor'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/scenario.rb:79:in `with_visitor'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:53:in `with_visitor'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:41:in `accept'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:64:in `block in visit_background'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:63:in `visit_background'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/feature.rb:36:in `accept'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:27:in `block in visit_feature'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:26:in `visit_feature'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:28:in `block in accept'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:27:in `accept'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:21:in `block in visit_features'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:20:in `visit_features'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/runtime.rb:49:in `run!'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/cli/main.rb:47:in `execute!'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/bin/cucumber:13:in `<top (required)>'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/bin/cucumber:23:in `load'
15:21:52       /mnt/home/jenkins-deploy/.gem/2.1.0/bin/cucumber:23:in `<main>'
15:21:52       tests/browser/features/issues.feature:7:in `And this page has issues'
15:21:53 
15:21:53   Scenario: Clicking page issues opens overlay     # tests/browser/features/issues.feature:9
15:21:53     When I click the page issues stamp             # tests/browser/features/step_definitions/issues_steps.rb:5
15:21:53     Then I should see the issues overlay           # tests/browser/features/step_definitions/issues_steps.rb:21
15:21:53 
15:21:53   Scenario: Closing page issues                    # tests/browser/features/issues.feature:13
15:21:53     When I click the page issues stamp             # tests/browser/features/step_definitions/issues_steps.rb:5
15:21:53     And I see the issues overlay                   # tests/browser/features/step_definitions/issues_steps.rb:9
15:21:53     And I click the overlay issue close button     # tests/browser/features/step_definitions/issues_steps.rb:1
15:21:53     Then I should not see the issues overlay       # tests/browser/features/step_definitions/issues_steps.rb:17
15:21:53 
15:21:53   Scenario: Closing page issues (browser back)     # tests/browser/features/issues.feature:19
15:21:53     When I click the page issues stamp             # tests/browser/features/step_definitions/issues_steps.rb:5
15:21:53     And I see the issues overlay                   # tests/browser/features/step_definitions/issues_steps.rb:9
15:21:53     And I click the browser back button            # tests/browser/features/step_definitions/common_steps.rb:68
15:21:53     Then I should not see the issues overlay       # tests/browser/features/step_definitions/issues_steps.rb:17
15:21:53

Developer notes

Saucelabs URIs would be helpful, but the issue here is that the new treatment for page issuest is currently running 50%of the time and when it does it fails.

The test should be updated to be compatible with the new treatment. The CSS selector can be updated to use both.

Replication steps

Locally:
Set up the A/B test and run the browser test.
50% of the time it fails.

Event Timeline

Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptOct 10 2018, 3:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Jdlrobson renamed this task from Page issues failing in CI to Prepare selenium browser test for page issues A/B test.Oct 10 2018, 5:18 PM
Jdlrobson updated the task description. (Show Details)

@phuedx, @ovasileva this would be a good task to pair with @Ryasmeen on

@phuedx, @ovasileva this would be a good task to pair with @Ryasmeen on

SYN-ACK. Thanks, @Jdlrobson!

ovasileva raised the priority of this task from Medium to High.Oct 16 2018, 4:31 PM

@Ryasmeen would you be interested in pairing on this today or tomorrow?

I'd use UBN if it didn't channel RelEng. This is causing a lot of noise in our Selenium jobs and we may be missing some important bugs.

Change 473307 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] QA: Page issues browser test is ready for new treatment

https://gerrit.wikimedia.org/r/473307

Jdlrobson lowered the priority of this task from High to Medium.Nov 16 2018, 6:32 PM

Also dropping to normal since the A/B test is not running so it's not causing us any pain right now...

Change 473307 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] QA: Page issues browser test is ready for new treatment

https://gerrit.wikimedia.org/r/473307

The last two builds (#706 and #705) have failed with the following error message:

[WARN] NotFound excludeKey exclude
excludes=[]
No emails were triggered.
Running YamlMatrixExecutionStrategy
Triggering selenium-MinervaNeue » chrome,beta,Linux,BrowserTests
Triggering selenium-MinervaNeue » firefox,beta,Linux,BrowserTests
Completed selenium-MinervaNeue » chrome,beta,Linux,BrowserTests FAILURE
Completed selenium-MinervaNeue » firefox,beta,Linux,BrowserTests FAILURE

I note that a large number of other builds failed for the same reason.

I've triggered a new build.

☝️ Mibad. The actual test failures are listed elsewhere. All three builds failed because the Beta Cluster is still in read-only mode.

Good news: I haven't noticed any test failures from pages issues in the last several builds.
Bad news: Other tests are failing and the beta cluster read only errors are making this hard to sign-off. I *think* it's working, but there is a lot of noise right now.

Should we do an optimistic sign off?

The page issues tests pass for me locally as shown below. Unfortunately, the tests run on the beta cluster are still flakey (and probably will be for the foreseeable future).

tests.png (784×2 px, 313 KB)

I'm signing off on this for now but will be watching the BC tests closely and will pull this back in I notice the page issues tests failing.