Page MenuHomePhabricator

Wikidata browser test jobs fail since upgrading to mediawiki-selenium 1.2.1
Closed, ResolvedPublic

Description

All Firefox Wikidata tests affected:
https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/

The test run well, but the job fails with
ERROR: Publisher hudson.tasks.junit.JUnitResultArchiver aborted due to exception
hudson.AbortException: None of the test reports contained any result

Event Timeline

WMDE-Fisch raised the priority of this task from to Needs Triage.
WMDE-Fisch updated the task description. (Show Details)
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 15 2015, 8:49 AM
zeljkofilipin triaged this task as High priority.Jun 15 2015, 10:44 AM

@WMDE-Fisch: if I do not manage to fix the problem by tomorrow afternoon our time, I will mention it at the next Browser-Tests triage meeting: https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Meetings/Notes

hashar renamed this task from jenkins jobs fail since upgrading to mediawiki-selenium 1.2.1 to Wikidata browser test jobs fail since upgrading to mediawiki-selenium 1.2.1.Jun 15 2015, 12:41 PM
dduvall added a comment.EditedJun 18 2015, 4:50 PM

Looks like something in the Raita logger is affecting the JUnit logger. Zeljko and I just verified this by commenting out the export RAITA_URL line in the build and it's all green.

https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/284/

Looks like the fix implemented for T100904 caused a regression. Unfortunately, all loggers share the same options object, so when the Raita logger modifies it to include the :expand option, that negatively impacts the JUnit logger as well.

After some more investigation, this appears to be a problem with the JUnit logger where scenario outlines are omitted from the XML when Cucumber's --expand option is enabled, so it's not strictly an issue with the Raita logger but an upstream bug—though Raita causes the side effect by forcing the --expand option.

The simplest case for reproducing this issue within the WikidataBrowserTests repo is to run the tests with the following invocation:

bundle exec cucumber --expand --format junit --out log/junit features/performance_test.feature:20

Inspecting the contents of log/junit/TEST-features-performance_test.xml shows that there's no <testcase> entry for the scenario outline at features/performance_test.feature:20 and no <testcase> elements at all since that's the only feature element being tested, which causes Jenkins to choke.

<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="0" failures="0" name="High performance" skipped="0" tests="0" time="0.000000">
  <system-out>
    <![CDATA[]]>
  </system-out>
  <system-err>
    <![CDATA[]]>
  </system-err>
</testsuite>

There's an upstream issue for this already and it's fixed in Cucumber 2.0. I might try to backport the fix to the 1.3 branch so we don't have to brave the major upgrade—from what I can tell it was a major overhaul and likely a bit unstable for now.

Backporting the expand/outline fix was a huge pain so I submitted a PR for a simple workaround: don't write out the XML if no testcases were logged.

https://github.com/cucumber/cucumber/pull/872

The PR has been merged. I'll upgrade our Cucumber dependency for the next release of MW-Selenium.

https://github.com/cucumber/cucumber-ruby/pull/872

Change 219882 had a related patch set uploaded (by Dduvall):
Upgrade cucumber dependency for fix to JUnit logger

https://gerrit.wikimedia.org/r/219882

Change 219882 merged by jenkins-bot:
Upgrade cucumber dependency for fix to JUnit logger

https://gerrit.wikimedia.org/r/219882

dduvall closed this task as Resolved.Jul 7 2015, 3:12 PM

The updated dependency is included in the 1.4 release.