For some reason, this jenkins job failed, but it's sauce labs job is set to passed.
This should not happen. If a jenkins job fails, it should also set it's sauce labs job to failed. Investigate if this is the only occurrence, or if there is more, and what the problem is.
All tests on QuickSurveys cannot be debugged because of this - https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | leila | T112326 [Epic] Reader segmentation research - Q2 | |||
| Resolved | • jhobs | T113443 First QuickSurvey for reader segmentation research - external survey | |||
| Resolved | Jdlrobson | T111445 GOAL: Run our first survey campaign | |||
| Resolved | Jdlrobson | T113534 Investigate QuickSurveys browser tests failures | |||
| Resolved | • dduvall | T105589 MW-Selenium associates wrong SauceLabs job with Jenkins artifact |
This keeps happening and now seems to account for all our failing builds. Two recent examples:
Any ideas what's going on?
Change 229417 had a related patch set uploaded (by Dduvall):
Test SauceLabs job status annotation
This seems to be becoming more and more of a problem.
I can't debug any of the failures quickly in:
https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/220/testReport/
:(
Any updates on this. We're getting spammed by emails and have no idea how to fix them as we don't know what's going on.
Do we know what's happening here? It's preventing us debugging a problem with browser tests.
@Jdlrobson: unfortunately, nobody had the time to take a look yet :(
I have added the task to the meeting notes for the next weekly triage.
@Jdlrobson, try upgrading to mediawiki_selenium 1.6.3 and see if this still occurs—check the upgrade docs for details on upgrading. The implementation for Sauce session ID tracking and URL construction was decoupled from Cucumber in https://gerrit.wikimedia.org/r/#/c/232660/ (>= 1.6.0) and may have incidentally fixed this bug.
@dduval https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/ is running 1.6.3 and has the same issue :/
I was finally able to reproduce this by introducing failures into the features/sauce.feature MW-Selenium tests. It appears that the order in which Cucumber executes its After hooks (where MW-S teardown happens which includes SauceLabs job annotation) and its formatter teardown is not fully deterministic. In my testing, the first scenario in a feature file results in the teardown consistently being evaluated before the formatter (resulting in the right annotation) but following scenarios result in the opposite.
I'll look into how this can be implemented without using the glorified-global-state garbage that it's using now.
Change 265894 had a related patch set uploaded (by Dduvall):
Log SauceLabs session URLs via Cucumber logger embeds
Change 265894 merged by jenkins-bot:
Log SauceLabs session URLs via Cucumber logger embeds
We have agreed to release a new version of mediawiki-selenium and bump the version requirements in repositories.
Change 266737 had a related patch set uploaded (by Hashar):
Bump mediawiki_selenium to 1.6.4
So we have bumped mediawiki_selenium from 1.6.3 to 1.6.4 as well as its dependencies.
The integration tests are proposed in https://gerrit.wikimedia.org/r/#/c/260881/ They passed with selenium 1.6.3 as seen on https://integration.wikimedia.org/ci/job/mwext-mw-selenium/3166/consoleFull
But a recheck with 1.6.4 fails ( https://integration.wikimedia.org/ci/job/mwext-mw-selenium/3185/consoleFull ) Lot of tests are skipped and a few fail with:
Background: # features/mmv.download.feature:4
Given I am viewing an image using MMV # features/step_definitions/mmv_steps.rb:21
undefined method `session_id' for #<Selenium::WebDriver::Driver:0x00000002366148> (NoMethodError)
./features/step_definitions/mmv_steps.rb:5:in `/^I am at a wiki article with at least two embedded pictures$/'
./features/step_definitions/mmv_steps.rb:22:in `/^I am viewing an image using MMV$/'
features/mmv.download.feature:5:in `Given I am viewing an image using MMV'I have retriggered one of the browser test https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/298/ . It has some tests passing, still running though.
Change 266765 had a related patch set uploaded (by Hashar):
Revert update of mediawiki_selenium dependencies
So I crafted a change that revert the selenium-webdriver dependencies change (i.e. only update 1.6.3 to 1.6.4) https://gerrit.wikimedia.org/r/#/c/266765/ I made it to depends on the change adding the integration test ( https://gerrit.wikimedia.org/r/#/c/260881/ ) using Depends-On.
Result is it fails equally. So seems either 1.6.4 is broken or the currently proposed integration tests are not compatible with 1.6.4. I am reverting the version bump.
Full stack trace from a scenario failing on https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/298/console
00:00:46.058 Feature: Download menu 00:00:46.059 00:00:46.059 Background: # features/mmv.download.feature:4 00:01:00.801 Given I am viewing an image using MMV # features/step_definitions/mmv_steps.rb:21 00:01:00.801 undefined method `session_id' for #<Selenium::WebDriver::Driver:0x565e3654489433c browser=:firefox> (NoMethodError) 00:01:00.801 ./features/step_definitions/mmv_steps.rb:5:in `/^I am at a wiki article with at least two embedded pictures$/' 00:01:00.802 ./features/step_definitions/mmv_steps.rb:22:in `/^I am viewing an image using MMV$/' 00:01:00.802 features/mmv.download.feature:5:in `Given I am viewing an image using MMV' 00:01:03.469
Feature: Multimedia Viewer performance
Background: # features/mmv.performance.feature:4
Given I am using a custom user agent # features/step_definitions/mmv_performance_steps.rb:26
undefined method `test_name' for #<MediawikiSelenium::Environment:0x000000041bdd58> (NoMethodError)
.gem/2.0.0/gems/rspec-expectations-2.99.2/lib/rspec/matchers/method_missing.rb:9:in `method_missing'
/mnt/jenkins-workspace/workspace/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/tests/browser/features/step_definitions/mmv_performance_steps.rb:27:in `block in <top (required)>'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `instance_exec'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `block in cucumber_instance_exec'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:69:in `cucumber_run_with_backtrace_filtering'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:36:in `cucumber_instance_exec'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/rb_support/rb_step_definition.rb:97:in `invoke'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/step_match.rb:25:in `invoke'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:60:in `invoke'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:38:in `accept'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:106:in `block in visit_step'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:105:in `visit_step'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:19:in `block in accept'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `each'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `accept'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:100:in `block in visit_steps'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:99:in `visit_steps'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:44:in `block in accept'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:54:in `block in with_visitor'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/scenario.rb:79:in `with_visitor'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:53:in `with_visitor'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:41:in `accept'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:64:in `block in visit_background'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:63:in `visit_background'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/feature.rb:36:in `accept'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:27:in `block in visit_feature'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:26:in `visit_feature'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:28:in `block in accept'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:27:in `accept'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:21:in `block in visit_features'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:20:in `visit_features'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/runtime.rb:49:in `run!'
.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/cli/main.rb:47:in `execute!'
.gem/2.0.0/gems/cucumber-1.3.20/bin/cucumber:13:in `<top (required)>'
.gem/2.0.0/bin/cucumber:23:in `load'
.gem/2.0.0/bin/cucumber:23:in `<main>'
features/mmv.performance.feature:5:in `Given I am using a custom user agent'
And I am at a wiki article with at least two embedded pictures # features/step_definitions/mmv_steps.rb:3Change 266766 had a related patch set uploaded (by Hashar):
Revert "Bump mediawiki_selenium to 1.6.4"
Reverted. I have recheck https://gerrit.wikimedia.org/r/#/c/260881/10 and triggered https://integration.wikimedia.org/ci/job/mwext-mw-selenium/3189/ . Seems to work again now.
I haven't verified, should be solved by mediawiki_selenium 1.6.4 (one really want >= 1.6.5 though).