Page MenuHomePhabricator

MW-Selenium associates wrong SauceLabs job with Jenkins artifact
Closed, ResolvedPublic

Description

For some reason, this jenkins job failed, but it's sauce labs job is set to passed.

This should not happen. If a jenkins job fails, it should also set it's sauce labs job to failed. Investigate if this is the only occurrence, or if there is more, and what the problem is.

All tests on QuickSurveys cannot be debugged because of this - https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/

Event Timeline

zeljkofilipin raised the priority of this task from to Needs Triage.
zeljkofilipin updated the task description. (Show Details)
Jdlrobson edited projects, added Readers-Web-Backlog; removed Gather.
Jdlrobson set Security to None.

Change 229417 had a related patch set uploaded (by Dduvall):
Test SauceLabs job status annotation

https://gerrit.wikimedia.org/r/229417

dduvall triaged this task as Medium priority.Aug 5 2015, 5:13 PM
dduvall moved this task from Inbox to Ruby on the Browser-Tests-Infrastructure board.

Change 229417 merged by jenkins-bot:
Test SauceLabs job status annotation

https://gerrit.wikimedia.org/r/229417

Any updates on this. We're getting spammed by emails and have no idea how to fix them as we don't know what's going on.

zeljkofilipin raised the priority of this task from Medium to High.Sep 30 2015, 3:19 PM

Do we know what's happening here? It's preventing us debugging a problem with browser tests.

@Jdlrobson: unfortunately, nobody had the time to take a look yet :(

I have added the task to the meeting notes for the next weekly triage.

@Jdlrobson, try upgrading to mediawiki_selenium 1.6.3 and see if this still occurs—check the upgrade docs for details on upgrading. The implementation for Sauce session ID tracking and URL construction was decoupled from Cucumber in https://gerrit.wikimedia.org/r/#/c/232660/ (>= 1.6.0) and may have incidentally fixed this bug.

dduvall renamed this task from Failed Jenkins job sets Sauce Labs job to passed to MW-Selenium associates wrong SauceLabs job with Jenkins artifact.Jan 22 2016, 9:24 PM

I was finally able to reproduce this by introducing failures into the features/sauce.feature MW-Selenium tests. It appears that the order in which Cucumber executes its After hooks (where MW-S teardown happens which includes SauceLabs job annotation) and its formatter teardown is not fully deterministic. In my testing, the first scenario in a feature file results in the teardown consistently being evaluated before the formatter (resulting in the right annotation) but following scenarios result in the opposite.

I'll look into how this can be implemented without using the glorified-global-state garbage that it's using now.

Change 265894 had a related patch set uploaded (by Dduvall):
Log SauceLabs session URLs via Cucumber logger embeds

https://gerrit.wikimedia.org/r/265894

Change 265894 merged by jenkins-bot:
Log SauceLabs session URLs via Cucumber logger embeds

https://gerrit.wikimedia.org/r/265894

We have agreed to release a new version of mediawiki-selenium and bump the version requirements in repositories.

Change 266737 had a related patch set uploaded (by Hashar):
Bump mediawiki_selenium to 1.6.4

https://gerrit.wikimedia.org/r/266737

Change 266737 merged by jenkins-bot:
Bump mediawiki_selenium to 1.6.4

https://gerrit.wikimedia.org/r/266737

So we have bumped mediawiki_selenium from 1.6.3 to 1.6.4 as well as its dependencies.

The integration tests are proposed in https://gerrit.wikimedia.org/r/#/c/260881/ They passed with selenium 1.6.3 as seen on https://integration.wikimedia.org/ci/job/mwext-mw-selenium/3166/consoleFull

But a recheck with 1.6.4 fails ( https://integration.wikimedia.org/ci/job/mwext-mw-selenium/3185/consoleFull ) Lot of tests are skipped and a few fail with:

Background:                             # features/mmv.download.feature:4
  Given I am viewing an image using MMV # features/step_definitions/mmv_steps.rb:21
    undefined method `session_id' for #<Selenium::WebDriver::Driver:0x00000002366148> (NoMethodError)
    ./features/step_definitions/mmv_steps.rb:5:in `/^I am at a wiki article with at least two embedded pictures$/'
    ./features/step_definitions/mmv_steps.rb:22:in `/^I am viewing an image using MMV$/'
    features/mmv.download.feature:5:in `Given I am viewing an image using MMV'

I have retriggered one of the browser test https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/298/ . It has some tests passing, still running though.

Might be due to:

-    selenium-webdriver (2.48.1)
+    selenium-webdriver (2.49.0)

Change 266765 had a related patch set uploaded (by Hashar):
Revert update of mediawiki_selenium dependencies

https://gerrit.wikimedia.org/r/266765

So I crafted a change that revert the selenium-webdriver dependencies change (i.e. only update 1.6.3 to 1.6.4) https://gerrit.wikimedia.org/r/#/c/266765/ I made it to depends on the change adding the integration test ( https://gerrit.wikimedia.org/r/#/c/260881/ ) using Depends-On.

Result is it fails equally. So seems either 1.6.4 is broken or the currently proposed integration tests are not compatible with 1.6.4. I am reverting the version bump.

Full stack trace from a scenario failing on https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/298/console

00:00:46.058 Feature: Download menu
00:00:46.059 
00:00:46.059   Background:                             # features/mmv.download.feature:4
00:01:00.801     Given I am viewing an image using MMV # features/step_definitions/mmv_steps.rb:21
00:01:00.801       undefined method `session_id' for #<Selenium::WebDriver::Driver:0x565e3654489433c browser=:firefox> (NoMethodError)
00:01:00.801       ./features/step_definitions/mmv_steps.rb:5:in `/^I am at a wiki article with at least two embedded pictures$/'
00:01:00.802       ./features/step_definitions/mmv_steps.rb:22:in `/^I am viewing an image using MMV$/'
00:01:00.802       features/mmv.download.feature:5:in `Given I am viewing an image using MMV'
00:01:03.469
Feature: Multimedia Viewer performance

  Background:                                                      # features/mmv.performance.feature:4
    Given I am using a custom user agent                           # features/step_definitions/mmv_performance_steps.rb:26
      undefined method `test_name' for #<MediawikiSelenium::Environment:0x000000041bdd58> (NoMethodError)
      .gem/2.0.0/gems/rspec-expectations-2.99.2/lib/rspec/matchers/method_missing.rb:9:in `method_missing'
      /mnt/jenkins-workspace/workspace/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/tests/browser/features/step_definitions/mmv_performance_steps.rb:27:in `block in <top (required)>'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `instance_exec'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `block in cucumber_instance_exec'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:69:in `cucumber_run_with_backtrace_filtering'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:36:in `cucumber_instance_exec'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/rb_support/rb_step_definition.rb:97:in `invoke'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/step_match.rb:25:in `invoke'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:60:in `invoke'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:38:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:106:in `block in visit_step'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:105:in `visit_step'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:19:in `block in accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `each'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:100:in `block in visit_steps'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:99:in `visit_steps'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:44:in `block in accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:54:in `block in with_visitor'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/scenario.rb:79:in `with_visitor'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:53:in `with_visitor'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:41:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:64:in `block in visit_background'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:63:in `visit_background'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/feature.rb:36:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:27:in `block in visit_feature'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:26:in `visit_feature'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:28:in `block in accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:27:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:21:in `block in visit_features'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:20:in `visit_features'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/runtime.rb:49:in `run!'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/cli/main.rb:47:in `execute!'
      .gem/2.0.0/gems/cucumber-1.3.20/bin/cucumber:13:in `<top (required)>'
      .gem/2.0.0/bin/cucumber:23:in `load'
      .gem/2.0.0/bin/cucumber:23:in `<main>'
      features/mmv.performance.feature:5:in `Given I am using a custom user agent'
    And I am at a wiki article with at least two embedded pictures # features/step_definitions/mmv_steps.rb:3

Change 266766 had a related patch set uploaded (by Hashar):
Revert "Bump mediawiki_selenium to 1.6.4"

https://gerrit.wikimedia.org/r/266766

Change 266766 merged by jenkins-bot:
Revert "Bump mediawiki_selenium to 1.6.4"

https://gerrit.wikimedia.org/r/266766

Change 266765 abandoned by Hashar:
Revert update of mediawiki_selenium dependencies

https://gerrit.wikimedia.org/r/266765

I think this is fixed. Please reopen if there is something left to be done here.

I haven't verified, should be solved by mediawiki_selenium 1.6.4 (one really want >= 1.6.5 though).