Page MenuHomePhabricator

Figure out how to accurately backfill MediaWiki core test code coverage data
Open, LowPublic

Description

Jenkins only stores build histories for ~30 days, so any history past that has been deleted AFAIK. We need to backfill this data to get a good idea of what the historical trend is and was for test code coverage data.

I very naively tried to run coverage in a custom docker container (https://gitlab.com/legoktm/tool-coverage/tree/master/docker) however the results I got were one percent off (a significant difference) what I should have been getting, probably due to environment differences (jessie vs stretch, php5 vs php7, etc.).

I think it's better if we just figure out how to run the Jenkins code coverage job, but against historical versions of MediaWiki core. I believe this should be possible via the parameterized builds feature. I'm envisioning a script that determines which core commit to run against, triggers a jenkins build against that commit, waits until it finishes, grabs the clover.xml file, and repeat going back further in history until it stops working.

Event Timeline

Legoktm created this task.Dec 13 2017, 2:58 AM
greg awarded a token.Dec 13 2017, 5:02 AM
Legoktm triaged this task as Low priority.Dec 28 2017, 11:00 AM

I'm not really convinced this is worth spending time on any more. My current investigation into coverage shows that historical data would mostly be skewed by lack of @covers tags. I think it would still be neat to have this data for the sake of having a complete graph, but I'm not planning to spend time on it.

Legoktm closed this task as Declined.Jan 3 2018, 8:20 AM

I disagree from myself from last week that this data would be neat - it would just be wrong. :(

Legoktm reopened this task as Open.Jan 22 2018, 10:29 PM

Addshore has been collecting this data! https://docs.google.com/spreadsheets/d/1mYZYE93WFidLchbu_gKjVMzz3ZEf_kAX9FLh1gbs-jE/pub?gid=2 Though his only covers includes/

Yes, only for /includes, but from 2013, which is basically when we started running daily coverage and using covers tags.