https://github.com/jenkinsci/xunit-plugin/pull/57 has been merged now, and will presumably be released as part of 1.105. Once that happens, we can move away from the custom 1.103-wmf.1 build I made that cherry-picked that patch and go back to tracking upstream releases.
Description
Description
Related Objects
Related Objects
- Duplicates Merged Here
- T253337: Evaluate if we still need our local fork of the jenkins xunit-plugin
Event Timeline
Comment Actions
This was actually released in xunit-2.0.0, so someone will need to look a bit closer to see what (if any) breaking changes were made before upgrading.
Comment Actions
Our fork is marked 1.103-wmf.1. Patches are listed at https://gerrit.wikimedia.org/r/#/q/project:+integration/jenkinsci/xunit-plugin and we just have @Legoktm patch https://gerrit.wikimedia.org/r/#/c/integration/jenkinsci/xunit-plugin/+/425965/ Added skipped tag for phpunit xml.
Latest upstream is 2.3.5 - https://plugins.jenkins.io/xunit
The breaking change seems to be the introduction of a schema validation. The default is to just support Ant Junit and Maven Surefire output format. We can probably assume that PHPUnit and others use the same format. Anyway 2.0.0 had the patch included:
- JENKINS-42715 xUnit does not support PHPUnit 5.x skipped attribute (nfalco79 to ray)
So I guess it is safe to say PHPUnit produced output will work fine. I guess we should just upgrade the plugin then monitor.
Comment Actions
That broke some reports. The debian glue jobs run lintian and has a script to generate a junit xml such as:
<testsuite time="0" tests="1" failures="0" skipped="0" errors="0" assertions="0"> <testcase name="lintian-checks" classname="lintian.<all>" assertions="0" time="0"/> </testsuite>
Which leads to:
INFO: Processing JUnit INFO: [JUnit] - 1 test report file(s) were found with the pattern '*lintian.xml' relative to '/srv/jenkins/workspace/workspace/debian-glue-backports' for the testing framework 'JUnit'. WARNING: The file '/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml' is an invalid file. WARNING: At line 2 of file:/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml:cvc-complex-type.3.2.2: Attribute 'assertions' is not allowed to appear in element 'testsuite'. WARNING: At line 2 of file:/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml:cvc-complex-type.4: Attribute 'name' must appear on element 'testsuite'. WARNING: At line 3 of file:/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml:cvc-complex-type.3.2.2: Attribute 'assertions' is not allowed to appear in element 'testcase'. ERROR: Step ‘Publish xUnit test result report’ failed: The result file '/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml' for the metric 'JUnit' is not valid. The result file has been skipped. Archiving artifacts Finished: FAILURE
https://gerrit.wikimedia.org/r/#/c/operations/software/atskafka/+/599322/
Comment Actions
Mentioned in SAL (#wikimedia-releng) [2020-05-28T13:31:24Z] <hashar> Rolling back Xunit Jenkins plugin # T194096
Comment Actions
I should have mentioned it. 2.0.0 is a breaking change, the plugin now passes the xml through schema validation https://plugins.jenkins.io/xunit/
Turns out the script we use to convert lintian output to junit is not recognized / invalid :/
Comment Actions
The Jenkins xunit plugin embeds a bunch of XSD files https://github.com/jenkinsci/xunit-plugin/tree/master/src/main/resources/org/jenkinsci/plugins/xunit/types
With Xunit 2.0.0, they started validating the given XML files against the XSD, the default being junit. For other set of tools they are validating them as well then use an XSL to normalize the input to be Junit compatible (as I understand it).
The default XSD can be retrieved at https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd
So:
git clone https://github.com/jenkinsci/xunit-plugin xmllint \ --noout \ --schema ./xunit-plugin/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd \ lintian.xml
lintian.xml:2: element testsuite: Schemas validity error : Element 'testsuite', attribute 'assertions': The attribute 'assertions' is not allowed. lintian.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'name' is required but missing. lintian.xml:3: element testcase: Schemas validity error : Element 'testcase', attribute 'assertions': The attribute 'assertions' is not allowed. lintian.xml fails to validate
Or with the PHUnit one:
phpunit-4.0.xsd
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'name' is required but missing. lintian-orig.xml fails to validate
If add name='Name of suite' to the <testsuite> element, it validates just fine.
The lintian-junit-report script we use comes from jenkins-debian-glue. At the top it refers to http://windyroad.com.au/dl/Open%20Source/JUnit.xsd which states JUnit test result schema for the Apache Ant JUnit and JUnitReport tasks but that does not even validate against it:
$ xmllint --noout --schema JUnit.xsd lintian.xml lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite', attribute 'skipped': The attribute 'skipped' is not allowed. lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite', attribute 'assertions': The attribute 'assertions' is not allowed. lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'name' is required but missing. lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'timestamp' is required but missing. lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'hostname' is required but missing. lintian-orig.xml:3: element testcase: Schemas validity error : Element 'testcase': This element is not expected. Expected is ( properties ). lintian-orig.xml fails to validate
Comment Actions
https://github.com/mika/jenkins-debian-glue/pull/233 got merged by upstream with congratulations \o/ I am so happy.
Comment Actions
Have to review T194096#6172999 and see whether the PHPUnit scheme works but from a previous comment:
2.0.0 had the patch included:
JENKINS-42715 xUnit does not support PHPUnit 5.x skipped attribute (nfalco79 to ray)
I have fixed the Debian glue report
Seems like we just need to upgrade the plugin and monitor at this point. I am assigning it to myself so the task bubbles up.