I am basically giving up on coveralls support. I've added Appveyor support to the python library, using their mostly undocumented API, but it refuses to group Appveyor multiple builds together (using the same git hash) like it does for Travis. If I try to group them manually using the same build identifier in the data submission, all builds in all branches are grouped together into one ever growing build which references the initial commit.

Anyway, a sample of the coveralls.io output.

https://coveralls.io/builds/2766042

While debugging their crappy service, as I noted on T101807, I found that they are not an open source kinda company, so it isnt easy to find and fix problems.

What I did find is an open source equivalent which Wikimedia could install
https://github.com/localytics/shamer

And I am giving the lesser known https://codecov.io a try, and I am quite impressed so far. Their github has more stuff in it, and they also offer their product for closed environments: https://github.com/codecov/enterprise - it looks very empty, so maybe it is closed source also.

But https://codecov.io/ provides a public service as well, and they have a lovely interface, and lots of useful features.

Most importantly, they merge multiple coverage reports for the same git hash together, even merging Appveyor (Win32) and Travis (Linux), which means we can reach 100% without adding # pragma: nocover to the code for branches which are versions / platform specific.

One feature that I like is the coverage reports can be filtered on codecov.io , so the builds can send all coverage data, and the default view would only include pywikibot/ and maybe scripts/ , but we can still look at data related to other parts of the codebase on our personal accounts by removing some of the filters.

Without further ado, here it is https://codecov.io/github/jayvdb/pywikibot-core/commits

Also worth looking at are the suggestions : https://codecov.io/github/jayvdb/pywikibot-core/features/suggestions?ref=codecov , which are currently drowned in scripts/ so I have excluded scripts using a filter.

Note that Appveyor builds didnt send data for that build, due to coverage not collecting the data. (should be fixed now, but waiting for Appveyor builds to run again).

Change 240564 had a related patch set uploaded (by Ladsgroup):
Add coverage in travis builds

https://gerrit.wikimedia.org/r/240564

@jayvdb codecov.io is a reasonable choice. Can we work on this?

To provide accurate coverage, all of the jobs run on Travis CI (using nose and without) and Appveyor need to be included. Not including Appveyor would be acceptable, but not including the non-nose jobs is unacceptable, as it would result in extremely misleading data given how many tests the SITE_ONLY job skip.

Appveyor support was finally merged in August. https://github.com/codecov/codecov-python/pull/32
It needed some additional improvements to be seamless, but it works very well now.
I've submitted a pull request to incorporate it into the appveyor demo project: https://github.com/ogrisel/python-appveyor-demo/pull/32

Just invoking codecov does not work.

An area that still needs to be fleshed out is the suboptimal arrangement that we send raw coverage data to an external website and are not able to retrieve it.
Appveyor does expose the raw coverage data:
https://ci.appveyor.com/project/jayvdb/python-appveyor-demo/build/1.0.140/job/cos98rl1267hudmy/artifacts

Travis-CI has a similar functionality : http://docs.travis-ci.com/user/uploading-artifacts/
But it requires manually setting up an AWS S3 account. They offer free accounts, so I'm going to try that now to see how feasible it is.

Another approach is for the Travis script to push the coverage data to a dedicated github repo, or even a gist, so that it is archived in case it is needed.
For example we could create a repo called something like pywikibot/coverage-data , and the push would work for any one who has commit access to that repo. An environment variable could override that repo identifier, for anyone who doesnt have commit access to that shared repo.

In T74863#1707770, @jayvdb wrote:

Travis-CI has a similar functionality : http://docs.travis-ci.com/user/uploading-artifacts/
But it requires manually setting up an AWS S3 account. They offer free accounts, so I'm going to try that now to see how feasible it is.

So, despite using their 'free' offering, I am billed AUD 0.11 for uploading a single build, consisting of 20 .coverage files.
https://travis-ci.org/jayvdb/pywikibot-core/builds/84076536

Maybe I set this up wrong, but it seems too simple to be charged, so I dont think this is an appropriate solution.

Another approach is for the Travis script to push the coverage data to a dedicated github repo, or even a gist, so that it is archived in case it is needed.
For example we could create a repo called something like pywikibot/coverage-data , and the push would work for any one who has commit access to that repo. An environment variable could override that repo identifier, for anyone who doesnt have commit access to that shared repo.

So, maybe we do this?
Any other approaches? anonymous dpaste?

In T74863#1710698, @jayvdb wrote:

In T74863#1707770, @jayvdb wrote:

Travis-CI has a similar functionality : http://docs.travis-ci.com/user/uploading-artifacts/
But it requires manually setting up an AWS S3 account. They offer free accounts, so I'm going to try that now to see how feasible it is.

So, despite using their 'free' offering, I am billed AUD 0.11 for uploading a single build, consisting of 20 .coverage files.
https://travis-ci.org/jayvdb/pywikibot-core/builds/84076536

To break this down, the billing costs for uploading one set of artefacts:

US East (Northern Virginia) Region 	Usage  	 
  Amazon Simple Storage Service Requests-Tier1
    $0.00 per request - PUT, COPY, POST, or LIST requests under the monthly global free tier 	2,000 Requests 	0.00
    $0.005 per 1,000 PUT, COPY, POST, or LIST requests 	14,627 Requests 	0.07
    Total: 	0.07
    Region Total: 	0.07
US West (Oregon) Region 	  	 
  Amazon Simple Storage Service USW2-Requests-Tier1
    $0.005 per 1,000 PUT, COPY, POST, or LIST requests 	28 Requests 	0.01
    Total: 	0.01
  Amazon Simple Storage Service USW2-Requests-Tier2
    $0.00 per request - GET and all other requests under the monthly global free tier 	69 Requests 	0.00

Maybe I set this up wrong, but it seems too simple to be charged, so I dont think this is an appropriate solution.

As far as I can see, the costs are because one set of artifacts exceeds the 'free' usage.

We have an epic task to provide pre merge code coverage report T101544: Provide (pre-merge) code coverage reports on patchsets

That is blocked on T101545: Provide infrastructure to store files by project/branch post-merge to compare with pre-merge. Potentially we could use OpenStack Swift which is already used to hold files for the wiki projects.

We need one for the Beta-Cluster-Infrastructure to match production T64835: Setup a Swift cluster on beta-cluster to match production. Quoting @Andrew (labs ops) the last update was in November 2014:

To support Swift in labs I want to allow keystone/swift authentication for service users so that we can have project- or tool-wide swift accounts. This requires adding a second ldap backend to keystone, and multiple keystone auth backends was broken in Havana.

So as I understand it the idea is to have labs provide a per project Swift container where we will be able to push build artifacts such as logs / reports. Zuul has the support to pass Swift related parameters to jobs.

Unfortunately while Pywikibot is not allowed to run all of its tests in Jenkins, and cant run Windows jobs in Jenkins, T101544 wont be especially useful for Pywikibot. It would provide useful results sometimes, and we should move more of our tests to using mock'd servers and host environments.

An object store like Swift would be useful if we can push coverage data into it from Travis and Appveyor.
Once we have the coverage files from various hosts, they can be combined with coverage combine.

On top of that we could run a (slightly customised) shamer, but that doesnt add much value IMO.

What looks more interesting is SonarQube, which can be integrated with Jenkins , as well as loading coverage data from files.
https://github.com/SonarSource/sonar-examples/tree/master/projects/languages/python/python-sonar-runner-coverage

Change 240564 merged by jenkins-bot:
Add coverage in travis builds

https://gerrit.wikimedia.org/r/240564

Two other options for online storage of .coverage data is:

• https://bintray.com/
• gh-pages branch of the same repo, or a different branch name

@Ricordisamoa: I guess your dislike has the same reason as T101807: Run Pywikibot tests against Win32 using Appveyor? In which case I strongly agree with @jayvdb's response (T101807#1355749).

I've raised the SaaS 'issue' upstream as https://github.com/codecov/support/issues/106

Worth noting here, we can obtain the coverage data for a specific Travis job via the codecov.io API: e.g. https://codecov.io/api/github/wikimedia/pywikibot-core/scripts/newitem.py?ref=a2ff58f485277a9b4d3ed90e1b9f8d2ea4654239&build=3059.11
It is in a different JSON format, but it should be trivial to covert that into format supported by other tools like gcov.

p.s. the Appveyor changeset still hasnt been merged. :/

In T74863#1765202, @jayvdb wrote:

I've raised the SaaS 'issue' upstream as https://github.com/codecov/support/issues/106

codecov has added support for downloading the raw data which was sent to it.

https://codecov.io/gh/wikimedia/pywikibot-core/commit/c6d89a03a1fc50559ac531ff8c5c70ef0a5c5b1b/builds

With regards to subprocesses, I am surprised that using pytest-cov doesnt appear to be automatically solving the problem.
Ideally we solve this problem in the upstream libraries.
If not, the two documented solutions are at http://coverage.readthedocs.io/en/coverage-4.0.3/subprocess.html

Change 290147 had a related patch set uploaded (by John Vandenberg):
Run codecov on CI job failure

https://gerrit.wikimedia.org/r/290147

Change 290147 merged by jenkins-bot:
Run codecov on CI job failure

https://gerrit.wikimedia.org/r/290147

In T74863#2316571, @jayvdb wrote:

With regards to subprocesses, I am surprised that using pytest-cov doesnt appear to be automatically solving the problem.
Ideally we solve this problem in the upstream libraries.
If not, the two documented solutions are at http://coverage.readthedocs.io/en/coverage-4.0.3/subprocess.html

With failed jobs now being submitted to codecov, it looks like script_tests subprocess coverage is being reported.

https://codecov.io/gh/wikimedia/pywikibot-core/tree/b8bee741d7ebb736b5039d078b90258c66d25848