Page MenuHomePhabricator

Track test code coverage long term
Open, Needs TriagePublic

Description

To get an idea of how we're doing, we need to track test code coverage long term. Jenkins only stores history for about 30 days.

See https://tools.wmflabs.org/coverage/mediawiki/chart.png as a proof of concept, which is generated via clover.xml files that aren't ever deleted. It currently has all of the history that jenkins had ~3 days ago and grabs each day's new coverage run.

Source code is currently at https://gitlab.com/legoktm/tool-coverage (will be moved to Phab once striker is fixed)

Event Timeline

Legoktm created this task.Dec 13 2017, 2:49 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 13 2017, 2:49 AM
Legoktm added a subscriber: hashar.Jan 3 2018, 8:16 AM

Here's my proposal:

We store clover.xml files in /srv/coverage on contint1001, and each repository gets its own subdirectory, based on the Gerrit name: /srv/coverage/mediawiki/core, /srv/coverage/mediawiki/extensions/Linter, and so on.

The clover.xml files are named YYYY-MM-DD-sha1.xml. For MediaWiki core, we'll store the bzip2 compressed files with the same name and an additional ".bz2" extension.

Any thoughts @hashar ?

See T101545: Provide infrastructure to store files by project/branch post-merge to compare with pre-merge for my request for this for all repos (as part of T101544: Provide (pre-merge) code coverage reports on patchsets when talking this through with @Krinkle a few years ago). This would be a nice step on that route.

Legoktm renamed this task from Track MediaWiki core test code coverage long term to Track test code coverage long term.Jan 4 2018, 11:45 PM
Legoktm updated the task description. (Show Details)

Ah! Didn't realize those were already filed as tickets. I clarified the bug name that this is intended for all repositories. And that is my ultimate goal - to provide pre-merge reports :)

Would this by chance be an approach to storing other CI data longer terms as well (i.e., test results)?

Would this by chance be an approach to storing other CI data longer terms as well (i.e., test results)?

The directory layout yes, but I'm not sure what other data we'd want to store. We use the checkstyle.xml format for PHPUnit/Phan/PHPCS/etc. but when everything passes, those XML files are empty, since there are no errors/warnings.

Other things we might want to record over time would be anything where 100%-or-it's-not-merged isn't the threshold:

  • Performance tests ("this method use to take 1.2ms and now it's 1.7ms");
  • Per-browser tests passing ("last week 97% of the unit tests passed in Edge, now these four are failing"); or
  • Per-DB tests passing ("could you please stop breaking postgres support?")

Etc.?

Hmm, maybe /srv/reports/mediawiki/core/coverage/... as the directory structure for this, so you could use /srv/reports/mediawiki/core/performance/, etc. in the future?

Hmm, maybe /srv/reports/mediawiki/core/coverage/... as the directory structure for this, so you could use /srv/reports/mediawiki/core/performance/, etc. in the future?

Might want to normalise slashes in repo names somehow (e.g. convert to dashes) to increase predictability and scannability of the directory structure (e.g. for indexing or browsing in some way). As well as to avoid conflicts between (parts of) a repo name and the report name. Especially given that the report names ("performance", "coverage") are quite plausible names for our repos (some already exist).

That sounds good to me (repo_name.replace('/', '-')).

greg awarded a token.Jan 10 2018, 12:22 AM
greg added a subscriber: greg.
Tgr added a subscriber: Tgr.Apr 15 2019, 6:55 AM

Is there / will there be an API to access this data?
One potential use case would be T220939: Display extension test coverage in infobox.

hashar removed a subscriber: hashar.Apr 15 2019, 4:04 PM
Krinkle removed a subscriber: Krinkle.Apr 15 2019, 9:18 PM
WDoranWMF added a subscriber: WDoranWMF.

@Jrbranaa We don't think CPT owns this task, could you advise if Code Health is the correct place to direct it?

@WDoranWMF, I do think this is a Code Health topic. I've added the Code-Health project tag to it so that it shows up on the Code Health Group's radar.