Page MenuHomePhabricator

add coverage report to tests
Closed, ResolvedPublic

Description

We should track what percentage of the codebase is being tested, probably on Travis.

coverage works:

$ coverage-2.7 run --source=pywikibot setup.py test

http://nose.readthedocs.org/en/latest/plugins/cover.html also works (it uses coverage):

$ nosetests --with-coverage --cover-package=pywikibot tests

I've uploaded our current coverage stats to

https://www.mediawiki.org/wiki/Manual:Pywikibot/Test_coverage

Coverage data is being sent to https://codecov.io/github/wikimedia/pywikibot for the Travis builds.

Still to do:

  • coverage of subprocesses
  • Appveyor (win32) builds
  • Jenkins/tox integration
  • coverage of failed jobs is usually still important data; incidental errors/failures in tests shouldnt cause our coverage percentages to decrease.
  • send coverage data to a location where it is accessible (https://github.com/codecov/support/issues/106)
  • periodically backup coverage data to a location under wikimedia control, so rough historical progress stats can be rebuilt later if necessary
  • Travis coverage data should include generate_* used before the tests.
  • Exclude .eggs from coverage data

Details

Reference
bz72863

Event Timeline

bzimport raised the priority of this task from to Needs Triage.
bzimport set Reference to bz72863.
bzimport added a subscriber: Unknown Object (????).
jayvdb created this task.Nov 1 2014, 2:26 PM
jayvdb moved this task from Backlog to Framework on the Pywikibot-tests board.Nov 29 2014, 4:57 PM

I am basically giving up on coveralls support. I've added Appveyor support to the python library, using their mostly undocumented API, but it refuses to group Appveyor multiple builds together (using the same git hash) like it does for Travis. If I try to group them manually using the same build identifier in the data submission, all builds in all branches are grouped together into one ever growing build which references the initial commit.

Anyway, a sample of the coveralls.io output.

https://coveralls.io/builds/2766042

While debugging their crappy service, as I noted on T101807, I found that they are not an open source kinda company, so it isnt easy to find and fix problems.

What I did find is an open source equivalent which Wikimedia could install
https://github.com/localytics/shamer

And I am giving the lesser known https://codecov.io a try, and I am quite impressed so far. Their github has more stuff in it, and they also offer their product for closed environments: https://github.com/codecov/enterprise - it looks very empty, so maybe it is closed source also.

But https://codecov.io/ provides a public service as well, and they have a lovely interface, and lots of useful features.

Most importantly, they merge multiple coverage reports for the same git hash together, even merging Appveyor (Win32) and Travis (Linux), which means we can reach 100% without adding # pragma: nocover to the code for branches which are versions / platform specific.

One feature that I like is the coverage reports can be filtered on codecov.io , so the builds can send all coverage data, and the default view would only include pywikibot/ and maybe scripts/ , but we can still look at data related to other parts of the codebase on our personal accounts by removing some of the filters.

Without further ado, here it is https://codecov.io/github/jayvdb/pywikibot-core/commits

Also worth looking at are the suggestions : https://codecov.io/github/jayvdb/pywikibot-core/features/suggestions?ref=codecov , which are currently drowned in scripts/ so I have excluded scripts using a filter.

Note that Appveyor builds didnt send data for that build, due to coverage not collecting the data. (should be fixed now, but waiting for Appveyor builds to run again).

Change 240564 had a related patch set uploaded (by Ladsgroup):
Add coverage in travis builds

https://gerrit.wikimedia.org/r/240564

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2015, 8:36 PM
XZise added a subscriber: XZise.Sep 23 2015, 8:49 PM

@jayvdb codecov.io is a reasonable choice. Can we work on this?

jayvdb added a comment.Oct 7 2015, 4:02 AM

To provide accurate coverage, all of the jobs run on Travis CI (using nose and without) and Appveyor need to be included. Not including Appveyor would be acceptable, but not including the non-nose jobs is unacceptable, as it would result in extremely misleading data given how many tests the SITE_ONLY job skip.

Appveyor support was finally merged in August. https://github.com/codecov/codecov-python/pull/32
It needed some additional improvements to be seamless, but it works very well now.
I've submitted a pull request to incorporate it into the appveyor demo project: https://github.com/ogrisel/python-appveyor-demo/pull/32

Just invoking codecov does not work.

An area that still needs to be fleshed out is the suboptimal arrangement that we send raw coverage data to an external website and are not able to retrieve it.
Appveyor does expose the raw coverage data:
https://ci.appveyor.com/project/jayvdb/python-appveyor-demo/build/1.0.140/job/cos98rl1267hudmy/artifacts

Travis-CI has a similar functionality : http://docs.travis-ci.com/user/uploading-artifacts/
But it requires manually setting up an AWS S3 account. They offer free accounts, so I'm going to try that now to see how feasible it is.

Another approach is for the Travis script to push the coverage data to a dedicated github repo, or even a gist, so that it is archived in case it is needed.
For example we could create a repo called something like pywikibot/coverage-data , and the push would work for any one who has commit access to that repo. An environment variable could override that repo identifier, for anyone who doesnt have commit access to that shared repo.

Travis-CI has a similar functionality : http://docs.travis-ci.com/user/uploading-artifacts/
But it requires manually setting up an AWS S3 account. They offer free accounts, so I'm going to try that now to see how feasible it is.

So, despite using their 'free' offering, I am billed AUD 0.11 for uploading a single build, consisting of 20 .coverage files.
https://travis-ci.org/jayvdb/pywikibot-core/builds/84076536

Maybe I set this up wrong, but it seems too simple to be charged, so I dont think this is an appropriate solution.

Another approach is for the Travis script to push the coverage data to a dedicated github repo, or even a gist, so that it is archived in case it is needed.
For example we could create a repo called something like pywikibot/coverage-data , and the push would work for any one who has commit access to that repo. An environment variable could override that repo identifier, for anyone who doesnt have commit access to that shared repo.

So, maybe we do this?
Any other approaches? anonymous dpaste?

jayvdb added a comment.EditedOct 7 2015, 10:16 PM

Travis-CI has a similar functionality : http://docs.travis-ci.com/user/uploading-artifacts/
But it requires manually setting up an AWS S3 account. They offer free accounts, so I'm going to try that now to see how feasible it is.

So, despite using their 'free' offering, I am billed AUD 0.11 for uploading a single build, consisting of 20 .coverage files.
https://travis-ci.org/jayvdb/pywikibot-core/builds/84076536

To break this down, the billing costs for uploading one set of artefacts:

US East (Northern Virginia) Region 	Usage  	 
  Amazon Simple Storage Service Requests-Tier1
    $0.00 per request - PUT, COPY, POST, or LIST requests under the monthly global free tier 	2,000 Requests 	0.00
    $0.005 per 1,000 PUT, COPY, POST, or LIST requests 	14,627 Requests 	0.07
    Total: 	0.07
    Region Total: 	0.07
US West (Oregon) Region 	  	 
  Amazon Simple Storage Service USW2-Requests-Tier1
    $0.005 per 1,000 PUT, COPY, POST, or LIST requests 	28 Requests 	0.01
    Total: 	0.01
  Amazon Simple Storage Service USW2-Requests-Tier2
    $0.00 per request - GET and all other requests under the monthly global free tier 	69 Requests 	0.00

Maybe I set this up wrong, but it seems too simple to be charged, so I dont think this is an appropriate solution.

As far as I can see, the costs are because one set of artifacts exceeds the 'free' usage.

We have an epic task to provide pre merge code coverage report T101544: Provide (pre-merge) code coverage reports on patchsets

That is blocked on T101545: Provide infrastructure to store files by project/branch post-merge to compare with pre-merge. Potentially we could use OpenStack Swift which is already used to hold files for the wiki projects.

We need one for the Beta-Cluster-Infrastructure to match production T64835: Setup a Swift cluster on beta-cluster to match production. Quoting @Andrew (labs ops) the last update was in November 2014:

To support Swift in labs I want to allow keystone/swift authentication for service users so that we can have project- or tool-wide swift accounts. This requires adding a second ldap backend to keystone, and multiple keystone auth backends was broken in Havana.

So as I understand it the idea is to have labs provide a per project Swift container where we will be able to push build artifacts such as logs / reports. Zuul has the support to pass Swift related parameters to jobs.

Unfortunately while Pywikibot is not allowed to run all of its tests in Jenkins, and cant run Windows jobs in Jenkins, T101544 wont be especially useful for Pywikibot. It would provide useful results sometimes, and we should move more of our tests to using mock'd servers and host environments.

An object store like Swift would be useful if we can push coverage data into it from Travis and Appveyor.
Once we have the coverage files from various hosts, they can be combined with coverage combine.

On top of that we could run a (slightly customised) shamer, but that doesnt add much value IMO.

What looks more interesting is SonarQube, which can be integrated with Jenkins , as well as loading coverage data from files.
https://github.com/SonarSource/sonar-examples/tree/master/projects/languages/python/python-sonar-runner-coverage

Change 240564 merged by jenkins-bot:
Add coverage in travis builds

https://gerrit.wikimedia.org/r/240564

jayvdb updated the task description. (Show Details)Oct 10 2015, 12:09 AM
jayvdb set Security to None.
jayvdb updated the task description. (Show Details)
jayvdb updated the task description. (Show Details)Oct 10 2015, 12:25 AM
jayvdb updated the task description. (Show Details)Oct 10 2015, 12:55 AM
Ricordisamoa added a subscriber: Ricordisamoa.

Two other options for online storage of .coverage data is:

XZise added a comment.Oct 11 2015, 4:56 PM

@Ricordisamoa: I guess your dislike has the same reason as T101807: Run Pywikibot tests against Win32 using Appveyor? In which case I strongly agree with @jayvdb's response (T101807#1355749).

hashar updated the task description. (Show Details)Oct 15 2015, 8:10 AM

I've raised the SaaS 'issue' upstream as https://github.com/codecov/support/issues/106

Worth noting here, we can obtain the coverage data for a specific Travis job via the codecov.io API: e.g. https://codecov.io/api/github/wikimedia/pywikibot-core/scripts/newitem.py?ref=a2ff58f485277a9b4d3ed90e1b9f8d2ea4654239&build=3059.11
It is in a different JSON format, but it should be trivial to covert that into format supported by other tools like gcov.

p.s. the Appveyor changeset still hasnt been merged. :/

jayvdb updated the task description. (Show Details)May 22 2016, 8:55 PM
jayvdb added a subscriber: AbdealiJK.

I've raised the SaaS 'issue' upstream as https://github.com/codecov/support/issues/106

codecov has added support for downloading the raw data which was sent to it.

https://codecov.io/gh/wikimedia/pywikibot-core/commit/c6d89a03a1fc50559ac531ff8c5c70ef0a5c5b1b/builds

jayvdb updated the task description. (Show Details)May 22 2016, 9:06 PM

With regards to subprocesses, I am surprised that using pytest-cov doesnt appear to be automatically solving the problem.
Ideally we solve this problem in the upstream libraries.
If not, the two documented solutions are at http://coverage.readthedocs.io/en/coverage-4.0.3/subprocess.html

Change 290147 had a related patch set uploaded (by John Vandenberg):
Run codecov on CI job failure

https://gerrit.wikimedia.org/r/290147

Change 290147 merged by jenkins-bot:
Run codecov on CI job failure

https://gerrit.wikimedia.org/r/290147

With regards to subprocesses, I am surprised that using pytest-cov doesnt appear to be automatically solving the problem.
Ideally we solve this problem in the upstream libraries.
If not, the two documented solutions are at http://coverage.readthedocs.io/en/coverage-4.0.3/subprocess.html

With failed jobs now being submitted to codecov, it looks like script_tests subprocess coverage is being reported.

https://codecov.io/gh/wikimedia/pywikibot-core/tree/b8bee741d7ebb736b5039d078b90258c66d25848

Framawiki updated the task description. (Show Details)Jan 2 2018, 11:42 AM
Xqt closed this task as Resolved.Jan 11 2019, 8:51 AM
Xqt claimed this task.
Xqt added a subscriber: Xqt.

Coverage reports is published already with https://codecov.io/gh/wikimedia/pywikibot