Page MenuHomePhabricator

Switch back to upstream jenkins xunit plugin after PHPUnit fix is released
Open, MediumPublic

Description

https://github.com/jenkinsci/xunit-plugin/pull/57 has been merged now, and will presumably be released as part of 1.105. Once that happens, we can move away from the custom 1.103-wmf.1 build I made that cherry-picked that patch and go back to tracking upstream releases.

Event Timeline

Legoktm changed the task status from Open to Stalled.May 7 2018, 11:08 PM
Legoktm triaged this task as Lowest priority.
Legoktm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2018, 11:08 PM
Vvjjkkii renamed this task from Switch back to upstream jenkins xunit plugin after PHPUnit fix is released to rfdaaaaaaa.Jul 1 2018, 1:11 AM
Vvjjkkii changed the task status from Stalled to Open.
Vvjjkkii raised the priority of this task from Lowest to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
thcipriani renamed this task from rfdaaaaaaa to Switch back to upstream jenkins xunit plugin after PHPUnit fix is released.Jul 1 2018, 6:51 PM
thcipriani changed the task status from Open to Stalled.
thcipriani updated the task description. (Show Details)
CommunityTechBot lowered the priority of this task from High to Lowest.Jul 5 2018, 6:38 PM
Legoktm changed the task status from Stalled to Open.Oct 3 2018, 3:03 AM

This was actually released in xunit-2.0.0, so someone will need to look a bit closer to see what (if any) breaking changes were made before upgrading.

hashar added a subscriber: hashar.

Our fork is marked 1.103-wmf.1. Patches are listed at https://gerrit.wikimedia.org/r/#/q/project:+integration/jenkinsci/xunit-plugin and we just have @Legoktm patch https://gerrit.wikimedia.org/r/#/c/integration/jenkinsci/xunit-plugin/+/425965/ Added skipped tag for phpunit xml.

Latest upstream is 2.3.5 - https://plugins.jenkins.io/xunit

The breaking change seems to be the introduction of a schema validation. The default is to just support Ant Junit and Maven Surefire output format. We can probably assume that PHPUnit and others use the same format. Anyway 2.0.0 had the patch included:

  • JENKINS-42715 xUnit does not support PHPUnit 5.x skipped attribute (nfalco79 to ray)

So I guess it is safe to say PHPUnit produced output will work fine. I guess we should just upgrade the plugin then monitor.

hashar raised the priority of this task from Lowest to Medium.Aug 28 2019, 4:24 PM
hashar moved this task from Untriaged to Next on the Continuous-Integration-Infrastructure board.
hashar reopened this task as Open.May 28 2020, 1:29 PM

That broke some reports. The debian glue jobs run lintian and has a script to generate a junit xml such as:

<testsuite time="0" tests="1" failures="0" skipped="0" errors="0" assertions="0">
   <testcase name="lintian-checks" classname="lintian.<all>" assertions="0" time="0"/>
</testsuite>

Which leads to:

INFO: Processing JUnit
INFO: [JUnit] - 1 test report file(s) were found with the pattern '*lintian.xml' relative to '/srv/jenkins/workspace/workspace/debian-glue-backports' for the testing framework 'JUnit'.
WARNING: The file '/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml' is an invalid file.
WARNING: At line 2 of file:/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml:cvc-complex-type.3.2.2: Attribute 'assertions' is not allowed to appear in element 'testsuite'.
WARNING: At line 2 of file:/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml:cvc-complex-type.4: Attribute 'name' must appear on element 'testsuite'.
WARNING: At line 3 of file:/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml:cvc-complex-type.3.2.2: Attribute 'assertions' is not allowed to appear in element 'testcase'.
ERROR: Step ‘Publish xUnit test result report’ failed: The result file '/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml' for the metric 'JUnit' is not valid. The result file has been skipped.
Archiving artifacts
Finished: FAILURE

https://gerrit.wikimedia.org/r/#/c/operations/software/atskafka/+/599322/

Mentioned in SAL (#wikimedia-releng) [2020-05-28T13:31:24Z] <hashar> Rolling back Xunit Jenkins plugin # T194096

I should have mentioned it. 2.0.0 is a breaking change, the plugin now passes the xml through schema validation https://plugins.jenkins.io/xunit/

Turns out the script we use to convert lintian output to junit is not recognized / invalid :/

The Jenkins xunit plugin embeds a bunch of XSD files https://github.com/jenkinsci/xunit-plugin/tree/master/src/main/resources/org/jenkinsci/plugins/xunit/types

With Xunit 2.0.0, they started validating the given XML files against the XSD, the default being junit. For other set of tools they are validating them as well then use an XSL to normalize the input to be Junit compatible (as I understand it).

The default XSD can be retrieved at https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

So:

git clone https://github.com/jenkinsci/xunit-plugin
xmllint \
  --noout \
  --schema ./xunit-plugin/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd \
  lintian.xml
lintian.xml:2: element testsuite: Schemas validity error : Element 'testsuite', attribute 'assertions': The attribute 'assertions' is not allowed.
lintian.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'name' is required but missing.
lintian.xml:3: element testcase: Schemas validity error : Element 'testcase', attribute 'assertions': The attribute 'assertions' is not allowed.
lintian.xml fails to validate

Or with the PHUnit one:

phpunit-4.0.xsd
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'name' is required but missing.
lintian-orig.xml fails to validate

If add name='Name of suite' to the <testsuite> element, it validates just fine.


The lintian-junit-report script we use comes from jenkins-debian-glue. At the top it refers to http://windyroad.com.au/dl/Open%20Source/JUnit.xsd which states JUnit test result schema for the Apache Ant JUnit and JUnitReport tasks but that does not even validate against it:

$ xmllint --noout --schema JUnit.xsd lintian.xml 
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite', attribute 'skipped': The attribute 'skipped' is not allowed.
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite', attribute 'assertions': The attribute 'assertions' is not allowed.
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'name' is required but missing.
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'timestamp' is required but missing.
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'hostname' is required but missing.
lintian-orig.xml:3: element testcase: Schemas validity error : Element 'testcase': This element is not expected. Expected is ( properties ).
lintian-orig.xml fails to validate

Does that mean we need to get Debian upstream to fix lintian and release it?

https://github.com/mika/jenkins-debian-glue/pull/233 got merged by upstream with congratulations \o/ I am so happy.

Jdforrester-WMF removed Jdforrester-WMF as the assignee of this task.Wed, Jul 8, 6:35 PM