Page MenuHomePhabricator

MW-Selenium associates wrong SauceLabs job with Jenkins artifact
Closed, ResolvedPublic

Description

For some reason, this jenkins job failed, but it's sauce labs job is set to passed.

This should not happen. If a jenkins job fails, it should also set it's sauce labs job to failed. Investigate if this is the only occurrence, or if there is more, and what the problem is.

All tests on QuickSurveys cannot be debugged because of this - https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/

Event Timeline

zeljkofilipin raised the priority of this task from to Needs Triage.
zeljkofilipin updated the task description. (Show Details)
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 11 2015, 12:56 PM
Jdlrobson edited projects, added Readers-Web-Backlog; removed Gather.
Jdlrobson set Security to None.

Change 229417 had a related patch set uploaded (by Dduvall):
Test SauceLabs job status annotation

https://gerrit.wikimedia.org/r/229417

dduvall triaged this task as Normal priority.Aug 5 2015, 5:13 PM
dduvall moved this task from Inbox to Ruby on the Browser-Tests-Infrastructure board.

This seems to be becoming more and more of a problem.
I can't debug any of the failures quickly in:
https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/220/testReport/
:(

Change 229417 merged by jenkins-bot:
Test SauceLabs job status annotation

https://gerrit.wikimedia.org/r/229417

Any updates on this. We're getting spammed by emails and have no idea how to fix them as we don't know what's going on.

greg added a subscriber: greg.
zeljkofilipin raised the priority of this task from Normal to High.Sep 30 2015, 3:19 PM

Do we know what's happening here? It's preventing us debugging a problem with browser tests.

Jdlrobson updated the task description. (Show Details)Oct 9 2015, 4:16 PM

@Jdlrobson: unfortunately, nobody had the time to take a look yet :(

I have added the task to the meeting notes for the next weekly triage.

Thanks @zeljkofilipin for the update.

JSharma moved this task from Incoming to 2015-16 Q4 on the Readers-Web-Backlog board.

@Jdlrobson, try upgrading to mediawiki_selenium 1.6.3 and see if this still occurs—check the upgrade docs for details on upgrading. The implementation for Sauce session ID tracking and URL construction was decoupled from Cucumber in https://gerrit.wikimedia.org/r/#/c/232660/ (>= 1.6.0) and may have incidentally fixed this bug.

dduvall renamed this task from Failed Jenkins job sets Sauce Labs job to passed to MW-Selenium associates wrong SauceLabs job with Jenkins artifact.Jan 22 2016, 9:24 PM
dduvall claimed this task.Jan 22 2016, 10:06 PM
dduvall moved this task from CI to In Progress on the Browser-Tests-Infrastructure board.
dduvall added a comment.EditedJan 23 2016, 12:39 AM

I was finally able to reproduce this by introducing failures into the features/sauce.feature MW-Selenium tests. It appears that the order in which Cucumber executes its After hooks (where MW-S teardown happens which includes SauceLabs job annotation) and its formatter teardown is not fully deterministic. In my testing, the first scenario in a feature file results in the teardown consistently being evaluated before the formatter (resulting in the right annotation) but following scenarios result in the opposite.

I'll look into how this can be implemented without using the glorified-global-state garbage that it's using now.

Change 265894 had a related patch set uploaded (by Dduvall):
Log SauceLabs session URLs via Cucumber logger embeds

https://gerrit.wikimedia.org/r/265894

hashar added a subscriber: hashar.Jan 23 2016, 11:13 PM

Change 265894 merged by jenkins-bot:
Log SauceLabs session URLs via Cucumber logger embeds

https://gerrit.wikimedia.org/r/265894

We have agreed to release a new version of mediawiki-selenium and bump the version requirements in repositories.

Change 266737 had a related patch set uploaded (by Hashar):
Bump mediawiki_selenium to 1.6.4

https://gerrit.wikimedia.org/r/266737

Change 266737 merged by jenkins-bot:
Bump mediawiki_selenium to 1.6.4

https://gerrit.wikimedia.org/r/266737

So we have bumped mediawiki_selenium from 1.6.3 to 1.6.4 as well as its dependencies.

The integration tests are proposed in https://gerrit.wikimedia.org/r/#/c/260881/ They passed with selenium 1.6.3 as seen on https://integration.wikimedia.org/ci/job/mwext-mw-selenium/3166/consoleFull

But a recheck with 1.6.4 fails ( https://integration.wikimedia.org/ci/job/mwext-mw-selenium/3185/consoleFull ) Lot of tests are skipped and a few fail with:

Background:                             # features/mmv.download.feature:4
  Given I am viewing an image using MMV # features/step_definitions/mmv_steps.rb:21
    undefined method `session_id' for #<Selenium::WebDriver::Driver:0x00000002366148> (NoMethodError)
    ./features/step_definitions/mmv_steps.rb:5:in `/^I am at a wiki article with at least two embedded pictures$/'
    ./features/step_definitions/mmv_steps.rb:22:in `/^I am viewing an image using MMV$/'
    features/mmv.download.feature:5:in `Given I am viewing an image using MMV'

I have retriggered one of the browser test https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/298/ . It has some tests passing, still running though.

Might be due to:

-    selenium-webdriver (2.48.1)
+    selenium-webdriver (2.49.0)

Change 266765 had a related patch set uploaded (by Hashar):
Revert update of mediawiki_selenium dependencies

https://gerrit.wikimedia.org/r/266765

So I crafted a change that revert the selenium-webdriver dependencies change (i.e. only update 1.6.3 to 1.6.4) https://gerrit.wikimedia.org/r/#/c/266765/ I made it to depends on the change adding the integration test ( https://gerrit.wikimedia.org/r/#/c/260881/ ) using Depends-On.

Result is it fails equally. So seems either 1.6.4 is broken or the currently proposed integration tests are not compatible with 1.6.4. I am reverting the version bump.

Full stack trace from a scenario failing on https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/298/console

00:00:46.058 Feature: Download menu
00:00:46.059 
00:00:46.059   Background:                             # features/mmv.download.feature:4
00:01:00.801     Given I am viewing an image using MMV # features/step_definitions/mmv_steps.rb:21
00:01:00.801       undefined method `session_id' for #<Selenium::WebDriver::Driver:0x565e3654489433c browser=:firefox> (NoMethodError)
00:01:00.801       ./features/step_definitions/mmv_steps.rb:5:in `/^I am at a wiki article with at least two embedded pictures$/'
00:01:00.802       ./features/step_definitions/mmv_steps.rb:22:in `/^I am viewing an image using MMV$/'
00:01:00.802       features/mmv.download.feature:5:in `Given I am viewing an image using MMV'
00:01:03.469
Feature: Multimedia Viewer performance

  Background:                                                      # features/mmv.performance.feature:4
    Given I am using a custom user agent                           # features/step_definitions/mmv_performance_steps.rb:26
      undefined method `test_name' for #<MediawikiSelenium::Environment:0x000000041bdd58> (NoMethodError)
      .gem/2.0.0/gems/rspec-expectations-2.99.2/lib/rspec/matchers/method_missing.rb:9:in `method_missing'
      /mnt/jenkins-workspace/workspace/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/tests/browser/features/step_definitions/mmv_performance_steps.rb:27:in `block in <top (required)>'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `instance_exec'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:48:in `block in cucumber_instance_exec'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:69:in `cucumber_run_with_backtrace_filtering'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/core_ext/instance_exec.rb:36:in `cucumber_instance_exec'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/rb_support/rb_step_definition.rb:97:in `invoke'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/step_match.rb:25:in `invoke'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:60:in `invoke'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_invocation.rb:38:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:106:in `block in visit_step'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:105:in `visit_step'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:19:in `block in accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `each'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/step_collection.rb:18:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:100:in `block in visit_steps'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:99:in `visit_steps'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:44:in `block in accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:54:in `block in with_visitor'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/scenario.rb:79:in `with_visitor'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:53:in `with_visitor'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/background.rb:41:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:64:in `block in visit_background'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:63:in `visit_background'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/feature.rb:36:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:27:in `block in visit_feature'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:26:in `visit_feature'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:28:in `block in accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:17:in `each'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/features.rb:27:in `accept'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:21:in `block in visit_features'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/ast/tree_walker.rb:20:in `visit_features'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/runtime.rb:49:in `run!'
      .gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/cli/main.rb:47:in `execute!'
      .gem/2.0.0/gems/cucumber-1.3.20/bin/cucumber:13:in `<top (required)>'
      .gem/2.0.0/bin/cucumber:23:in `load'
      .gem/2.0.0/bin/cucumber:23:in `<main>'
      features/mmv.performance.feature:5:in `Given I am using a custom user agent'
    And I am at a wiki article with at least two embedded pictures # features/step_definitions/mmv_steps.rb:3

Change 266766 had a related patch set uploaded (by Hashar):
Revert "Bump mediawiki_selenium to 1.6.4"

https://gerrit.wikimedia.org/r/266766

Change 266766 merged by jenkins-bot:
Revert "Bump mediawiki_selenium to 1.6.4"

https://gerrit.wikimedia.org/r/266766

Change 266765 abandoned by Hashar:
Revert update of mediawiki_selenium dependencies

https://gerrit.wikimedia.org/r/266765

This is now solved, right?

zeljkofilipin closed this task as Resolved.Feb 23 2016, 3:42 PM

I think this is fixed. Please reopen if there is something left to be done here.

I haven't verified, should be solved by mediawiki_selenium 1.6.4 (one really want >= 1.6.5 though).