Page MenuHomePhabricator

mediawiki_selenium 1.2 breaks mw/core browser test when reporting to raita
Closed, ResolvedPublic

Description

mediawiki/core has been switched to use mediawiki-selenium 1.2 via https://gerrit.wikimedia.org/r/#/c/214413

I triggered a run of the browser test. They are all fine until the reporting is triggered
https://integration.wikimedia.org/ci/view/BrowserTests/view/Core/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/629/consoleFull

19 scenarios (19 passed)
100 steps (100 passed)
11m5.757s
undefined method `[]' for nil:NilClass (NoMethodError)
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/mediawiki_selenium-1.2.0/lib/mediawiki_selenium/raita/logger.rb:48:in `block (2 levels) in amend_feature'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/mediawiki_selenium-1.2.0/lib/mediawiki_selenium/raita/logger.rb:47:in `each'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/mediawiki_selenium-1.2.0/lib/mediawiki_selenium/raita/logger.rb:47:in `block in amend_feature'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/mediawiki_selenium-1.2.0/lib/mediawiki_selenium/raita/logger.rb:44:in `each'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/mediawiki_selenium-1.2.0/lib/mediawiki_selenium/raita/logger.rb:44:in `amend_feature'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/mediawiki_selenium-1.2.0/lib/mediawiki_selenium/raita/logger.rb:19:in `block in after_features'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/mediawiki_selenium-1.2.0/lib/mediawiki_selenium/raita/logger.rb:18:in `each'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/mediawiki_selenium-1.2.0/lib/mediawiki_selenium/raita/logger.rb:18:in `after_features'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/cucumber-1.3.19/lib/cucumber/ast/tree_walker.rb:181:in `block in send_to_all'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/cucumber-1.3.19/lib/cucumber/ast/tree_walker.rb:179:in `each'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/cucumber-1.3.19/lib/cucumber/ast/tree_walker.rb:179:in `send_to_all'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/cucumber-1.3.19/lib/cucumber/ast/tree_walker.rb:171:in `broadcast'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/cucumber-1.3.19/lib/cucumber/ast/tree_walker.rb:20:in `visit_features'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/cucumber-1.3.19/lib/cucumber/runtime.rb:49:in `run!'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/cucumber-1.3.19/lib/cucumber/cli/main.rb:47:in `execute!'
/mnt/jenkins-workspace/workspace/gems/2.0.0/gems/cucumber-1.3.19/bin/cucumber:13:in `<top (required)>'
/mnt/jenkins-workspace/workspace/gems/2.0.0/bin/cucumber:23:in `load'
/mnt/jenkins-workspace/workspace/gems/2.0.0/bin/cucumber:23:in `<main>'
+ echo -e '\nJob has failed (exit code: 1).'

Job has failed (exit code: 1).
+ false
Build step 'Execute shell' marked build as failure

Seems something is wrong in the raita reporter.

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
zeljkofilipin triaged this task as Unbreak Now! priority.Jun 2 2015, 3:55 PM
zeljkofilipin set Security to None.
zeljkofilipin moved this task from Inbox to Next on the Browser-Tests-Infrastructure board.

Cucumber's JSON formatter appears to omit results for each of the scenario outline examples.

https://github.com/cucumber/gherkin/issues/165

I'll see if I can implement a workaround to the Raita::Logger class.

It looks like we can work around this bug by forcing the --expand option in Raita::Logger which will include each outline example in the output as a separate scenario. One unfortunate side effect, however, is that the "pretty" console formatter will also have a more verbose style when outputting outline results—Cucumber passes around a single object reference for options to all the formatters so there's no clean way around this.

The output for test suites that make heavy use of outlines will be particularly bloated (e.g. CirrusSearch). (Keep in mind this side effect will only apply to CI runs.)

Without --expand:

@chrome @clean @firefox @phantomjs
Feature: Create account

  Scenario Outline: Go to Create account page   # features/create_account.feature:4
    Given I go to Create account page at <path> # features/step_definitions/create_account_steps.rb:12
    Then form has Create account button         # features/step_definitions/create_account_steps.rb:16

    Examples:
      | path                          |
      | Special:CreateAccount         |
      | Special:UserLogin/signup      |
      | Special:UserLogin?type=signup |

3 scenarios (3 passed)
6 steps (6 passed)
0m34.018s

With --expand:

@chrome @clean @firefox @phantomjs
Feature: Create account

  Scenario Outline: Go to Create account page   # features/create_account.feature:4
    Given I go to Create account page at <path> # features/step_definitions/create_account_steps.rb:12
    Then form has Create account button         # features/step_definitions/create_account_steps.rb:16

    Examples:

      Scenario: | Special:CreateAccount |   # features/create_account.feature:10
        Given I go to Create account page at Special:CreateAccount# features/step_definitions/create_account_steps.rb:12
        Then form has Create account button         # features/step_definitions/create_account_steps.rb:16

      Scenario: | Special:UserLogin/signup |   # features/create_account.feature:11
        Given I go to Create account page at Special:UserLogin/signup# features/step_definitions/create_account_steps.rb:12
        Then form has Create account button         # features/step_definitions/create_account_steps.rb:16

      Scenario: | Special:UserLogin?type=signup |   # features/create_account.feature:12
        Given I go to Create account page at Special:UserLogin?type=signup# features/step_definitions/create_account_steps.rb:12
        Then form has Create account button         # features/step_definitions/create_account_steps.rb:16

3 scenarios (3 passed)
6 steps (6 passed)
0m33.266s

If this change in output is acceptable, I can go ahead and implement the workaround.

Change 215501 had a related patch set uploaded (by Dduvall):
Force expanded formatter output for scenario outlines

https://gerrit.wikimedia.org/r/215501

Change 215501 merged by jenkins-bot:
Force expanded formatter output for scenario outlines

https://gerrit.wikimedia.org/r/215501

Change 215514 had a related patch set uploaded (by Dduvall):
QA: Upgrade mediawiki_selenium for fix to Raita logging

https://gerrit.wikimedia.org/r/215514

Change 215805 had a related patch set uploaded (by Dduvall):
QA: Upgrade mediawiki_selenium for fix to SauceJobs reporting

https://gerrit.wikimedia.org/r/215805

Change 215805 merged by jenkins-bot:
QA: Upgrade mediawiki_selenium for fix to SauceJobs reporting

https://gerrit.wikimedia.org/r/215805

Change 215514 merged by jenkins-bot:
QA: Upgrade mediawiki_selenium for fix to Raita logging

https://gerrit.wikimedia.org/r/215514

Solved in mediawiki_selenium 1.2.1

SamanthaNguyen added a subscriber: SamanthaNguyen.

Removing Patch-For-Review since this has already been merged and resolved