Page MenuHomePhabricator

Track test code coverage long term
Open, Needs TriagePublic

Description

To get an idea of how we're doing, we need to track test code coverage long term. Jenkins only stores history for about 30 days.

See https://tools.wmflabs.org/coverage/mediawiki/chart.png as a proof of concept, which is generated via clover.xml files that aren't ever deleted. It currently has all of the history that jenkins had ~3 days ago and grabs each day's new coverage run.

Source code is currently at https://gitlab.com/legoktm/tool-coverage (will be moved to Phab once striker is fixed)

Event Timeline

Here's my proposal:

We store clover.xml files in /srv/coverage on contint1001, and each repository gets its own subdirectory, based on the Gerrit name: /srv/coverage/mediawiki/core, /srv/coverage/mediawiki/extensions/Linter, and so on.

The clover.xml files are named YYYY-MM-DD-sha1.xml. For MediaWiki core, we'll store the bzip2 compressed files with the same name and an additional ".bz2" extension.

Any thoughts @hashar ?

Legoktm renamed this task from Track MediaWiki core test code coverage long term to Track test code coverage long term.Jan 4 2018, 11:45 PM
Legoktm updated the task description. (Show Details)

Ah! Didn't realize those were already filed as tickets. I clarified the bug name that this is intended for all repositories. And that is my ultimate goal - to provide pre-merge reports :)

Would this by chance be an approach to storing other CI data longer terms as well (i.e., test results)?

Would this by chance be an approach to storing other CI data longer terms as well (i.e., test results)?

The directory layout yes, but I'm not sure what other data we'd want to store. We use the checkstyle.xml format for PHPUnit/Phan/PHPCS/etc. but when everything passes, those XML files are empty, since there are no errors/warnings.

Other things we might want to record over time would be anything where 100%-or-it's-not-merged isn't the threshold:

  • Performance tests ("this method use to take 1.2ms and now it's 1.7ms");
  • Per-browser tests passing ("last week 97% of the unit tests passed in Edge, now these four are failing"); or
  • Per-DB tests passing ("could you please stop breaking postgres support?")

Etc.?

Hmm, maybe /srv/reports/mediawiki/core/coverage/... as the directory structure for this, so you could use /srv/reports/mediawiki/core/performance/, etc. in the future?

Hmm, maybe /srv/reports/mediawiki/core/coverage/... as the directory structure for this, so you could use /srv/reports/mediawiki/core/performance/, etc. in the future?

Might want to normalise slashes in repo names somehow (e.g. convert to dashes) to increase predictability and scannability of the directory structure (e.g. for indexing or browsing in some way). As well as to avoid conflicts between (parts of) a repo name and the report name. Especially given that the report names ("performance", "coverage") are quite plausible names for our repos (some already exist).

That sounds good to me (repo_name.replace('/', '-')).

Is there / will there be an API to access this data?
One potential use case would be T220939: Display extension test coverage in infobox.

WDoranWMF added a subscriber: WDoranWMF.

@Jrbranaa We don't think CPT owns this task, could you advise if Code Health is the correct place to direct it?

@WDoranWMF, I do think this is a Code Health topic. I've added the Code-Health project tag to it so that it shows up on the Code Health Group's radar.