Error in code coverage reports for extensions in codehealth-pipeline
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	kostajh
	Sep 6 2019, 10:23 AM

Description

The https://doc.wikimedia.org/cover-extensions/ resource is empty for extensions in the codehealth-pipeline. Those extensions were deliberately removed from extension-coverage to avoid duplicating resources and efforts (see T224990 and specifically 55043fb).

But it seems like the publisher is not working as intended. @thcipriani and I tested this and it was working when this was first deployed.

- mwext-codehealth-{name}:
         # Used for analyzing the master branch on postmerge.
         name: 'master-non-voting'
         concurrent: false
         branch: '$ZUUL_BRANCH'
         sonar_args: |
             -Dsonar.projectKey=${{ZUUL_PROJECT//\//-}} \
             -Dsonar.projectName=${{ZUUL_PROJECT//\//-}} \
             -Dsonar.organization=wmftest \
             -Dsonar.host.url=https://sonarcloud.io \
             -X
         publishers:
             - archive-log-allow-empty
             - castor-save-workspace-cache
             - postbuildscript:
                   script-only-if-succeeded: false
                   script-only-if-failed: false
                   builders:
                       -   cover-extensions-publish:
                               src: 'cover'
                               # Should be equivalent to $EXT_NAME
                               dest: '$DOC_BASENAME'
                       -   docker-wipe-dir:
                               dir: 'cover'
             - docker-cleanup

Related Objects

Mentioned In: T259109: mwext-codehealth-master-non-voting erase doc.wikimedia.org coverage report
Mentioned Here: T259109: mwext-codehealth-master-non-voting erase doc.wikimedia.org coverage report
T224990: Enable codehealth pipeline for more extensions
rCICF55043fbd4b96: Publish coverage reports from extension-codehealth

Event Timeline

kostajh created this task.Sep 6 2019, 10:23 AM

Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptSep 6 2019, 10:23 AM

kostajh changed the subtype of this task from "Task" to "Bug Report".Sep 6 2019, 10:23 AM

Ah, it's only failing for extensions which have no unit tests, and which have 0% coverage in the junit.xml. These then show up as having 100% coverage in doc.wikimedia.org/cover-extensions.

We didn't intend to change the doc.wikimedia.org/cover-extensions report, but by switching to using unit tests and not integration tests for coverage calculation, we have.

Some options:

We could add back extension-coverage to extensions in the pipeline at the expense of more CI resources
Do nothing
Add a step for mwext-codehealth-master-non-voting which re-runs the PHPUnit tests to generate coverage, including integration tests this time, and rsync those results to doc.wikimedia.org/cover-extensions

In T232195#5474581, @kostajh wrote:

Ah, it's only failing for extensions which have no unit tests, and which have 0% coverage in the junit.xml. These then show up as having 100% coverage in doc.wikimedia.org/cover-extensions.

We didn't intend to change the doc.wikimedia.org/cover-extensions report, but by switching to using unit tests and not integration tests for coverage calculation, we have.

Depends on what folks think is useful, I suppose. Unit test coverage vs coverage anywhere. Unintentionally breaking the old behavior seems bad though.

Showing 100% on the dashboard when that actually means 0% seems like a separate bug (correct to say that that was not introduced by moving pipelines?).

hashar merged a task: T259109: mwext-codehealth-master-non-voting erase doc.wikimedia.org coverage report.Aug 7 2020, 12:31 PM

hashar mentioned this in T259109: mwext-codehealth-master-non-voting erase doc.wikimedia.org coverage report.

hashar added subscribers: DannyS712, hashar, Legoktm, Jdforrester-WMF.

I propose:

we reintroduce the extension-coverage template which will be in charge of publishing to doc.wikimedia.org
stop publishing the report from the codehealth job since it is broken / empty in some case. Essentially revert 55043fbd4b96

We will wait for @kostajh who is in vacations (the issue has been around for a while so we can wait a few more days).

hashar added projects: Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)).Aug 7 2020, 12:35 PM

In T232195#6368305, @hashar wrote:

I propose:

we reintroduce the extension-coverage template which will be in charge of publishing to doc.wikimedia.org

stop publishing the report from the codehealth job since it is broken / empty in some case. Essentially revert 55043fbd4b96

We will wait for @kostajh who is in vacations (the issue has been around for a while so we can wait a few more days).

That seems OK to me. Maybe eventually we could try again, but for now the race condition errors, conflicts with Wikibase, and differing coverage sources (unit tests only vs unit + integration) seem less than ideal.

thcipriani edited projects, added Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)); removed Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)).Oct 21 2020, 1:07 AM

thcipriani removed a project: Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)).Apr 20 2021, 12:21 AM

thcipriani added a project: Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)).Apr 20 2021, 12:34 AM

thcipriani removed a project: Release-Engineering-Team (CI & Testing services).Apr 20 2021, 1:09 AM

thcipriani edited projects, added Release-Engineering-Team (Doing); removed Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)).Apr 20 2021, 4:26 AM