Page MenuHomePhabricator

Publish the codex code coverage report
Closed, ResolvedPublic

Description

Codex currently only requires 80% test coverage for branches, functions, and lines. I was hoping to raise this and add the tests for the uncovered code, but I couldn't seem to find a code coverage report. Can the report be published to https://doc.wikimedia.org/cover/ ? I understand that it is generated automatically by jest

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 772492 had a related patch set uploaded (by Catrope; author: Catrope):

[design/codex@main] build: Add "npm run coverage" command

https://gerrit.wikimedia.org/r/772492

Change 772493 had a related patch set uploaded (by Catrope; author: Catrope):

[integration/config@master] Zuul: [design/codex] Publish code coverage reports

https://gerrit.wikimedia.org/r/772493

As proposed this only publishes the test coverage report for the main library (@wikimedia/codex), not for the icons or docs packages. In the future we could work on trying to merge the coverage reports into one for publication, but for now the main library report provides most of the value. The icons package contains minimal code, and the docs package is internal (not published or used outside of Codex development).

As proposed this only publishes the test coverage report for the main library (@wikimedia/codex), not for the icons or docs packages. In the future we could work on trying to merge the coverage reports into one for publication, but for now the main library report provides most of the value. The icons package contains minimal code, and the docs package is internal (not published or used outside of Codex development).

I said this on one of the patches, but I'll repeat it here for visibility - I think it would be helpful to have reports for all of the packages. Since test coverage is generally not published widely and is just used for Codex development I think its okay to have coverage reports for the docs package.

I agree, I would like to publish coverage for all of our packages. I think that would require the following things:

  • Use a tool to merge all the coverage XML files into one (I think istanbul/nyc can do this)
  • Merge the LCOV (HTML) reports into one somehow, perhaps by:
    • Using some other tool to merge them; or
    • Regenerating the LCOV report from the merged XML file; or
    • Manually stringing them together with an index.html file that links to each package's LCOV report
  • Update the root package.json file so that "npm run coverage" does all of those things and puts the result in the coverage/ directory

I agree, I would like to publish coverage for all of our packages. I think that would require the following things:

  • Use a tool to merge all the coverage XML files into one (I think istanbul/nyc can do this)
  • Merge the LCOV (HTML) reports into one somehow, perhaps by:
    • Using some other tool to merge them; or
    • Regenerating the LCOV report from the merged XML file; or
    • Manually stringing them together with an index.html file that links to each package's LCOV report
  • Update the root package.json file so that "npm run coverage" does all of those things and puts the result in the coverage/ directory

what about just putting them in separate directories?

/coverage/codex -> components
/coverage/codex-docs -> docs
/coverage/codex-icons -> icons
/coverage/codex-design-tokens -> design tokens

at least as an initial option

Change 772492 merged by jenkins-bot:

[design/codex@main] build: Add "npm run coverage" command

https://gerrit.wikimedia.org/r/772492

what about just putting them in separate directories?

/coverage/codex -> components
/coverage/codex-docs -> docs
/coverage/codex-icons -> icons
/coverage/codex-design-tokens -> design tokens

at least as an initial option

Yes, I thought about that, and for the web view that would work (if you also add a basic index.html file that links to each of them). That was basically option 2C in my bullet list, I just didn't word it very well.

However, the coverage overview page at https://doc.wikimedia.org/cover/ also appears to expect an XML file that it appears to use to show the coverage total. I don't know what that UI would fall back to if the XML file is missing (right now there are no projects listed that don't have an XML file and an overall coverage percentage). We can try that and see what happens, or we can try to build a combined XML file.

However, the coverage overview page at https://doc.wikimedia.org/cover/ also appears to expect an XML file that it appears to use to show the coverage total. I don't know what that UI would fall back to if the XML file is missing (right now there are no projects listed that don't have an XML file and an overall coverage percentage). We can try that and see what happens, or we can try to build a combined XML file.

It should gracefully degrade to listing but not having a progress bar (if not, that's a bug :))

It should gracefully degrade to listing but not having a progress bar (if not, that's a bug :))

Good to know, thanks! I just wasn't sure because there aren't any current examples of it doing this, since every project listed there has an XML file.

With that information, I propose we:

Another option could be to have multiple CI jobs that run npm run coverage, npm run coverage-docs, npm run coverage-icons, etc. and then sync each to its own top level directory under doc.wm.o/cover/ (codex, codex-docs, etc.).

Another option could be to have multiple CI jobs that run npm run coverage, npm run coverage-docs, npm run coverage-icons, etc. and then sync each to its own top level directory under doc.wm.o/cover/ (codex, codex-docs, etc.).

Ooh, that's a great idea and would be much easier. It will require some tweaks to the CI config, where both "npm run coverage" and the directory path it pulls from are hard-coded, but I hope I can figure that out. Instead of separate npm run commands, I'd probably want to run "npm test" once, then pull the different coverage dirs from packages/*/coverage and publish each of those to a different URL. oojs-ui has a job that does multiple publishes like that, so I'll look to that for inspiration.

Change 772493 merged by jenkins-bot:

[integration/config@master] Zuul: [design/codex] Publish code coverage reports

https://gerrit.wikimedia.org/r/772493

Mentioned in SAL (#wikimedia-releng) [2022-03-24T15:00:38Z] <James_F> Zuul: [design/codex] Publish code coverage reports for T303899

Hmm, doesn't appear to be visible at https://doc.wikimedia.org/cover/ yet?

ping @Catrope - are you still working on this?

Hmm, doesn't appear to be visible at https://doc.wikimedia.org/cover/ yet?

@Jdforrester-WMF just attempted to fix this with rCICFf51dc108460f: Zuul: Don't use browser-direct-coverage where browser-coverage will do. However, it doesn't appear that that worked, because https://gerrit.wikimedia.org/r/c/design/codex/+/818175 did not run the coverage job when it merged.

Change 819702 had a related patch set uploaded (by Catrope; author: Jforrester):

[integration/config@master] Zuul: Run publish jobs on branches called 'main' too

https://gerrit.wikimedia.org/r/819702

Jdforrester-WMF renamed this task from Public codex code coverage report to Publish the codex code coverage report.Aug 3 2022, 2:46 PM

OK, so good news / bad news; the custom post-merge coverage job for Codex now runs, but it fails with 'error TS2307: Cannot find module '@wikimedia/codex-icons' or its corresponding type' and then a bunch of other errors. Possibly the coverage job needs a build first?

Change 820180 had a related patch set uploaded (by Catrope; author: Catrope):

[design/codex@main] build: Run build-all-if-missing in "npm coverage"

https://gerrit.wikimedia.org/r/820180

Change 820180 merged by jenkins-bot:

[design/codex@main] build: Run build-all-if-missing in "npm coverage"

https://gerrit.wikimedia.org/r/820180

Change 823725 had a related patch set uploaded (by Anne Tomasevich; author: Anne Tomasevich):

[mediawiki/core@master] Update Codex from v0.1.0-alpha.9 to v0.1.0-alpha.10

https://gerrit.wikimedia.org/r/823725

Change 823725 merged by jenkins-bot:

[mediawiki/core@master] Update Codex from v0.1.0-alpha.9 to v0.1.0-alpha.10

https://gerrit.wikimedia.org/r/823725