The tests that fail vary but 3 out of the last 5 failures have been page issue specific.
15:21:35 @integration @chrome @en.m.wikipedia.beta.wmflabs.org @firefox @test2.m.wikipedia.org @vagrant 15:21:35 Feature: Issues 15:21:35 15:21:35 Background: # tests/browser/features/issues.feature:4 15:21:52 Given I am using the mobile site # tests/browser/features/step_definitions/common_steps.rb:40 15:21:52 And I am on a page which has cleanup templates # tests/browser/features/step_definitions/create_page_api_steps.rb:30 15:21:52 And this page has issues # tests/browser/features/step_definitions/issues_steps.rb:13 15:21:52 timed out after 5 seconds, waiting for {:css=>".mw-mf-cleanup"} to become present (Watir::Wait::TimeoutError) 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/watir-webdriver-0.9.1/lib/watir-webdriver/wait.rb:44:in `until' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/watir-webdriver-0.9.1/lib/watir-webdriver/wait.rb:223:in `wait_until_present' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/page-object-1.1.0/lib/page-object/platforms/watir_webdriver/element.rb:160:in `when_present' 15:21:52 /srv/jenkins-workspace/workspace/selenium-MinervaNeue/BROWSER/chrome/MEDIAWIKI_ENVIRONMENT/beta/PLATFORM/Linux/label/BrowserTests/tests/browser/features/step_definitions/issues_steps.rb:14:in `block in <top (required)>' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `instance_exec' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `block in cucumber_instance_exec' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:69:in `cucumber_run_with_backtrace_filtering' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:36:in `cucumber_instance_exec' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/rb_support/rb_step_definition.rb:97:in `invoke' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/step_match.rb:25:in `invoke' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:60:in `invoke' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:38:in `accept' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:106:in `block in visit_step' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:105:in `visit_step' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:19:in `block in accept' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `each' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `accept' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:100:in `block in visit_steps' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:99:in `visit_steps' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:44:in `block in accept' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:54:in `block in with_visitor' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/scenario.rb:79:in `with_visitor' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:53:in `with_visitor' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:41:in `accept' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:64:in `block in visit_background' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:63:in `visit_background' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/feature.rb:36:in `accept' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:27:in `block in visit_feature' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:26:in `visit_feature' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:28:in `block in accept' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:27:in `accept' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:21:in `block in visit_features' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:20:in `visit_features' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/runtime.rb:49:in `run!' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/lib/cucumber/cli/main.rb:47:in `execute!' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/gems/cucumber-1.3.20/bin/cucumber:13:in `<top (required)>' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/bin/cucumber:23:in `load' 15:21:52 /mnt/home/jenkins-deploy/.gem/2.1.0/bin/cucumber:23:in `<main>' 15:21:52 tests/browser/features/issues.feature:7:in `And this page has issues' 15:21:53 15:21:53 Scenario: Clicking page issues opens overlay # tests/browser/features/issues.feature:9 15:21:53 When I click the page issues stamp # tests/browser/features/step_definitions/issues_steps.rb:5 15:21:53 Then I should see the issues overlay # tests/browser/features/step_definitions/issues_steps.rb:21 15:21:53 15:21:53 Scenario: Closing page issues # tests/browser/features/issues.feature:13 15:21:53 When I click the page issues stamp # tests/browser/features/step_definitions/issues_steps.rb:5 15:21:53 And I see the issues overlay # tests/browser/features/step_definitions/issues_steps.rb:9 15:21:53 And I click the overlay issue close button # tests/browser/features/step_definitions/issues_steps.rb:1 15:21:53 Then I should not see the issues overlay # tests/browser/features/step_definitions/issues_steps.rb:17 15:21:53 15:21:53 Scenario: Closing page issues (browser back) # tests/browser/features/issues.feature:19 15:21:53 When I click the page issues stamp # tests/browser/features/step_definitions/issues_steps.rb:5 15:21:53 And I see the issues overlay # tests/browser/features/step_definitions/issues_steps.rb:9 15:21:53 And I click the browser back button # tests/browser/features/step_definitions/common_steps.rb:68 15:21:53 Then I should not see the issues overlay # tests/browser/features/step_definitions/issues_steps.rb:17 15:21:53
Developer notes
Saucelabs URIs would be helpful, but the issue here is that the new treatment for page issuest is currently running 50%of the time and when it does it fails.
The test should be updated to be compatible with the new treatment. The CSS selector can be updated to use both.
Replication steps
Locally:
Set up the A/B test and run the browser test.
50% of the time it fails.