Page MenuHomePhabricator

Browser test jobs should use xUnit publisher instead of Junit
Closed, ResolvedPublic

Description

Our browser tests are configured using the Jenkins Junit plugin while others are using xUnit plugin.

The later as the ability to track regressions based on the previous build. Since browser tests jobs have a linear history, we could take advantage of that plugin. In theory, if a test failed in the previous build, that would not cause the job to FAIL (red) but simply be marked UNSTABLE (yellow). I think it will make the life of developers easier when tracking down tests breakages, got the idea following a mail by @Jdlrobson last week.

Have to test what happens exactly :-)

JJB documentation:

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added subscribers: hashar, Jdlrobson.
hashar triaged this task as Medium priority.Apr 3 2015, 12:04 PM
hashar moved this task from Untriaged to Next on the Continuous-Integration-Infrastructure board.

The only jobs that uses the xunit publisher are:

$ grep -l xunit /var/lib/jenkins/jobs/*/config.xml|cut -d\/ -f6
integration-phpunit-mediawiki-master
integration-phpunit-mediawiki-REL1_23
integration-phpunit-mediawiki-REL1_24
integration-phpunit-mediawiki-REL1_25
mediawiki-core-release
mediawiki-phpunit-hhvm-composer
mediawiki-phpunit-zend-composer
mediawiki-phpunit-zend
operations-mw-config-phpunit
operations-mw-config-tests
phpunit
test-hashar-zuul-debian-package
wikimedia-fundraising-civicrm

So maybe we want to migrate them to the JUnit publisher instead.

Comparing the two plugins, the output are exactly the same. Xunit has the ability to tweak thresholds though but for most jobs we are using the default values.

Example XML configuration snippets:

Junit
https://integration.wikimedia.org/ci/job/mediawiki-extensions-zend/

<hudson.tasks.junit.JUnitResultArchiver plugin="junit@1.6">
    <testResults>log/junit*.xml</testResults>
    <keepLongStdio>true</keepLongStdio>
    <testDataPublishers/>
</hudson.tasks.junit.JUnitResultArchiver>

xunit
https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/

<xunit plugin="xunit@1.96">
    <types>
        <PHPUnitJunitHudsonTestType>
            <pattern>log/junit-mw-phpunit.xml</pattern>
            <skipNoTestFiles>false</skipNoTestFiles>
            <failIfNotNew>true</failIfNotNew>
            <deleteOutputFiles>true</deleteOutputFiles>
            <stopProcessingIfError>true</stopProcessingIfError>
        </PHPUnitJunitHudsonTestType>
    </types>
    <thresholds/>
    <thresholdMode>1</thresholdMode>
    <extraConfiguration>
        <testTimeMargin>3000</testTimeMargin>
    </extraConfiguration>
</xunit>

In JJB all our xunit based macros looks like:

- xunit:
   types:
    - phpunit:
       pattern: 'log/junit-mw-phpunit.xml'

In the Web interface, a xunit based jobs has the following options checked:

  • Skip if there are no test files
  • Fail the build if test results were not updated this run
  • Delete temporary JUnit files
  • Stop and set the build to 'failed' status if there are errors when processing a result file

I am not sure whether the JUnit plugin has the same features.

zeljkofilipin lowered the priority of this task from Medium to Low.May 29 2017, 10:28 AM

@hashar do you still think this is something that needs to be done, or can this be resolved?

Change 365933 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Add unstable status to browser tests jobs

https://gerrit.wikimedia.org/r/365933

Change 365933 merged by jenkins-bot:
[integration/config@master] Add unstable status to browser tests jobs

https://gerrit.wikimedia.org/r/365933

zeljkofilipin moved this task from Backlog to In progress on the Continuous-Integration-Config board.

Resolved by https://gerrit.wikimedia.org/r/#/c/365933
Please reopen if there is anything left to do.

How can I see this in action?

I took a look at Selenium jobs. MultimediaViewer, RelatedArticles and Wikibase are failing.

RelatedArticles is failing with Could not locate Gemfile (T171847) so there are no failing tests. MultimediaViewer has failed just once in a row. Wikibase is failing for some time but every build has some new errors.

Conclusion: there are no builds that have failed twice in a row with the exactly same errors. Or, something is wrong. 😄