Maniphest T194096

Switch back to upstream jenkins xunit plugin after PHPUnit fix is released
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Legoktm
	May 7 2018, 11:08 PM

Description

https://github.com/jenkinsci/xunit-plugin/pull/57 has been merged now, and will presumably be released as part of 1.105. Once that happens, we can move away from the custom 1.103-wmf.1 build I made that cherry-picked that patch and go back to tracking upstream releases.

Details

	Project	Branch	Lines +/-	Subject
	integration/config	master	+1 -1	Upgrade jjb to 3.10.0

Customize query in gerrit

Related Objects

Mentioned In: T295719: beta-build-scap-deb failing
Mentioned Here: T256402: Remove JUnit artefacts from Quibble jobs

Event Timeline

Legoktm changed the task status from Open to Stalled.May 7 2018, 11:08 PM

Legoktm triaged this task as Lowest priority.

Legoktm created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2018, 11:08 PM

• Vvjjkkii renamed this task from Switch back to upstream jenkins xunit plugin after PHPUnit fix is released to rfdaaaaaaa.Jul 1 2018, 1:11 AM

• Vvjjkkii changed the task status from Stalled to Open.

• Vvjjkkii raised the priority of this task from Lowest to High.

• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii removed a subscriber: Aklapper.

thcipriani renamed this task from rfdaaaaaaa to Switch back to upstream jenkins xunit plugin after PHPUnit fix is released.Jul 1 2018, 6:51 PM

thcipriani changed the task status from Open to Stalled.

thcipriani updated the task description. (Show Details)

thcipriani removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.Jul 1 2018, 6:52 PM

thcipriani added a subscriber: Aklapper.

CommunityTechBot lowered the priority of this task from High to Lowest.Jul 5 2018, 6:38 PM

This was actually released in xunit-2.0.0, so someone will need to look a bit closer to see what (if any) breaking changes were made before upgrading.

Our fork is marked 1.103-wmf.1. Patches are listed at https://gerrit.wikimedia.org/r/#/q/project:+integration/jenkinsci/xunit-plugin and we just have @Legoktm patch https://gerrit.wikimedia.org/r/#/c/integration/jenkinsci/xunit-plugin/+/425965/ Added skipped tag for phpunit xml.

Latest upstream is 2.3.5 - https://plugins.jenkins.io/xunit

The breaking change seems to be the introduction of a schema validation. The default is to just support Ant Junit and Maven Surefire output format. We can probably assume that PHPUnit and others use the same format. Anyway 2.0.0 had the patch included:

JENKINS-42715 xUnit does not support PHPUnit 5.x skipped attribute (nfalco79 to ray)

So I guess it is safe to say PHPUnit produced output will work fine. I guess we should just upgrade the plugin then monitor.

hashar raised the priority of this task from Lowest to Medium.Aug 28 2019, 4:24 PM

hashar moved this task from Untriaged to Next on the Continuous-Integration-Infrastructure board.

hashar merged a task: T253337: Evaluate if we still need our local fork of the jenkins xunit-plugin.May 25 2020, 7:52 AM

hashar added a subscriber: Jdforrester-WMF.

Done. All seems well.

That broke some reports. The debian glue jobs run lintian and has a script to generate a junit xml such as:

<testsuite time="0" tests="1" failures="0" skipped="0" errors="0" assertions="0">
   <testcase name="lintian-checks" classname="lintian.<all>" assertions="0" time="0"/>
</testsuite>

Which leads to:

INFO: Processing JUnit
INFO: [JUnit] - 1 test report file(s) were found with the pattern '*lintian.xml' relative to '/srv/jenkins/workspace/workspace/debian-glue-backports' for the testing framework 'JUnit'.
WARNING: The file '/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml' is an invalid file.
WARNING: At line 2 of file:/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml:cvc-complex-type.3.2.2: Attribute 'assertions' is not allowed to appear in element 'testsuite'.
WARNING: At line 2 of file:/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml:cvc-complex-type.4: Attribute 'name' must appear on element 'testsuite'.
WARNING: At line 3 of file:/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml:cvc-complex-type.3.2.2: Attribute 'assertions' is not allowed to appear in element 'testcase'.
ERROR: Step ‘Publish xUnit test result report’ failed: The result file '/srv/jenkins/workspace/workspace/debian-glue-backports/atskafka-0.8+0~20200528130535.110+buster+wikimedia~1.gbpe8119f.lintian.xml' for the metric 'JUnit' is not valid. The result file has been skipped.
Archiving artifacts
Finished: FAILURE

https://gerrit.wikimedia.org/r/#/c/operations/software/atskafka/+/599322/

Mentioned in SAL (#wikimedia-releng) [2020-05-28T13:31:24Z] <hashar> Rolling back Xunit Jenkins plugin # T194096

I should have mentioned it. 2.0.0 is a breaking change, the plugin now passes the xml through schema validation https://plugins.jenkins.io/xunit/

Turns out the script we use to convert lintian output to junit is not recognized / invalid :/

The Jenkins xunit plugin embeds a bunch of XSD files https://github.com/jenkinsci/xunit-plugin/tree/master/src/main/resources/org/jenkinsci/plugins/xunit/types

With Xunit 2.0.0, they started validating the given XML files against the XSD, the default being junit. For other set of tools they are validating them as well then use an XSL to normalize the input to be Junit compatible (as I understand it).

The default XSD can be retrieved at https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

So:

git clone https://github.com/jenkinsci/xunit-plugin
xmllint \
  --noout \
  --schema ./xunit-plugin/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd \
  lintian.xml

lintian.xml:2: element testsuite: Schemas validity error : Element 'testsuite', attribute 'assertions': The attribute 'assertions' is not allowed.
lintian.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'name' is required but missing.
lintian.xml:3: element testcase: Schemas validity error : Element 'testcase', attribute 'assertions': The attribute 'assertions' is not allowed.
lintian.xml fails to validate

Or with the PHUnit one:

phpunit-4.0.xsd

lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'name' is required but missing.
lintian-orig.xml fails to validate

If add name='Name of suite' to the <testsuite> element, it validates just fine.

The lintian-junit-report script we use comes from jenkins-debian-glue. At the top it refers to http://windyroad.com.au/dl/Open%20Source/JUnit.xsd which states JUnit test result schema for the Apache Ant JUnit and JUnitReport tasks but that does not even validate against it:

$ xmllint --noout --schema JUnit.xsd lintian.xml 
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite', attribute 'skipped': The attribute 'skipped' is not allowed.
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite', attribute 'assertions': The attribute 'assertions' is not allowed.
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'name' is required but missing.
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'timestamp' is required but missing.
lintian-orig.xml:2: element testsuite: Schemas validity error : Element 'testsuite': The attribute 'hostname' is required but missing.
lintian-orig.xml:3: element testcase: Schemas validity error : Element 'testcase': This element is not expected. Expected is ( properties ).
lintian-orig.xml fails to validate

Does that mean we need to get Debian upstream to fix lintian and release it?

A quick and dirty fix is https://github.com/mika/jenkins-debian-glue/pull/233 :]

Jdforrester-WMF moved this task from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.Jun 3 2020, 5:10 PM

https://github.com/mika/jenkins-debian-glue/pull/233 got merged by upstream with congratulations \o/ I am so happy.

thcipriani edited projects, added Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)); removed Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)).Jul 8 2020, 5:34 PM

Jdforrester-WMF removed Jdforrester-WMF as the assignee of this task.Jul 8 2020, 6:35 PM

thcipriani edited projects, added Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)); removed Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)).Oct 21 2020, 1:07 AM

thcipriani removed a project: Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)).Apr 20 2021, 12:21 AM

thcipriani added a project: Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)).Apr 20 2021, 12:34 AM

thcipriani removed a project: Release-Engineering-Team (CI & Testing services).Apr 20 2021, 1:09 AM

thcipriani edited projects, added Release-Engineering-Team (Doing); removed Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)).Apr 20 2021, 4:26 AM

Have to review T194096#6172999 and see whether the PHPUnit scheme works but from a previous comment:

2.0.0 had the patch included:

JENKINS-42715 xUnit does not support PHPUnit 5.x skipped attribute (nfalco79 to ray)

I have fixed the Debian glue report

Seems like we just need to upgrade the plugin and monitor at this point. I am assigning it to myself so the task bubbles up.

thcipriani removed hashar as the assignee of this task.Jul 27 2021, 5:45 PM

thcipriani edited projects, added Release-Engineering-Team (Seen); removed Release-Engineering-Team (Doing).

Quibble based jobs no more generate PHPUnit Junit reports after T256402

Change 738911 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Upgrade jjb to 3.10.0

https://gerrit.wikimedia.org/r/738911

gerritbot added a project: Patch-For-Review.Nov 15 2021, 2:57 PM

Mentioned in SAL (#wikimedia-releng) [2021-11-15T14:59:00Z] <hashar> CI Jenkins: upgrading the Xunit plugin from 1.103 hack to 3.0.5 # T194096

hashar closed this task as Resolved.Nov 15 2021, 3:06 PM

hashar claimed this task.

hashar edited projects, added Release-Engineering-Team (Done by Wed 24 Nov 🧟); removed Patch-For-Review, Release-Engineering-Team (Seen).Nov 15 2021, 3:11 PM

Mentioned in SAL (#wikimedia-releng) [2021-11-15T16:37:59Z] <hashar> Upgraded Jenkins Job Builder to 3.10.0 which update support for the xunit plugin (got it updated to 3.x) # T194096

Change 738911 merged by jenkins-bot: