To get an idea of how we're doing, we need to track test code coverage long term. Jenkins only stores history for about 30 days.
See https://tools.wmflabs.org/coverage/mediawiki/chart.png as a proof of concept, which is generated via clover.xml files that aren't ever deleted. It currently has all of the history that jenkins had ~3 days ago and grabs each day's new coverage run.
Source code is currently at https://gitlab.com/legoktm/tool-coverage (will be moved to Phab once striker is fixed)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T182749 Track test code coverage long term | |||
| Open | None | T182750 Figure out how to accurately backfill MediaWiki core test code coverage data | |||
| Open | None | T182751 Migrate https://tools.wmflabs.org/coverage/mediawiki/ to CI infrastructure |
Here's my proposal:
We store clover.xml files in /srv/coverage on contint1001, and each repository gets its own subdirectory, based on the Gerrit name: /srv/coverage/mediawiki/core, /srv/coverage/mediawiki/extensions/Linter, and so on.
The clover.xml files are named YYYY-MM-DD-sha1.xml. For MediaWiki core, we'll store the bzip2 compressed files with the same name and an additional ".bz2" extension.
Any thoughts @hashar ?
See T101545: Provide infrastructure to store files by project/branch post-merge to compare with pre-merge for my request for this for all repos (as part of T101544: Provide (pre-merge) code coverage reports on patchsets when talking this through with @Krinkle a few years ago). This would be a nice step on that route.
Ah! Didn't realize those were already filed as tickets. I clarified the bug name that this is intended for all repositories. And that is my ultimate goal - to provide pre-merge reports :)
Would this by chance be an approach to storing other CI data longer terms as well (i.e., test results)?
The directory layout yes, but I'm not sure what other data we'd want to store. We use the checkstyle.xml format for PHPUnit/Phan/PHPCS/etc. but when everything passes, those XML files are empty, since there are no errors/warnings.
Other things we might want to record over time would be anything where 100%-or-it's-not-merged isn't the threshold:
Etc.?
Hmm, maybe /srv/reports/mediawiki/core/coverage/... as the directory structure for this, so you could use /srv/reports/mediawiki/core/performance/, etc. in the future?
Might want to normalise slashes in repo names somehow (e.g. convert to dashes) to increase predictability and scannability of the directory structure (e.g. for indexing or browsing in some way). As well as to avoid conflicts between (parts of) a repo name and the report name. Especially given that the report names ("performance", "coverage") are quite plausible names for our repos (some already exist).
Is there / will there be an API to access this data?
One potential use case would be T220939: Display extension test coverage in infobox.
@Jrbranaa We don't think CPT owns this task, could you advise if Code Health is the correct place to direct it?
@WDoranWMF, I do think this is a Code Health topic. I've added the Code-Health project tag to it so that it shows up on the Code Health Group's radar.