Page MenuHomePhabricator

py.test doesnt support selecting tests using test object attributes
Closed, ResolvedPublic

Description

Pywikibot uses nose attributes to select only tests that run on a certain site. This is used for the Travis-CI SITE_ONLY=1 jobs, so they run less unit tests on a specific host only in a short period of time.

py.test doesnt have any similar functionality. As it is a nose plugin, it should probably be re-implemented as a py.test plugin , and py.test says they merge any very useful plugins into the core product once it is stable.

Event Timeline

jayvdb created this task.May 18 2016, 5:30 PM
Restricted Application added subscribers: pywikibot-bugs-list, Zppix, Aklapper. · View Herald TranscriptMay 18 2016, 5:30 PM
jayvdb renamed this task from py.test doesnt support selecting tests using method attributes to py.test doesnt support selecting tests using test object attributes.May 18 2016, 5:33 PM

pytest does have an equivalent pytest.mark, but it seems to be a little limited. It can only be used by doing:

  • It can be used with the decorator pytest.mark.NAME
  • It can be used by declaring the pytestmark attribute like pytestmark = [pytest.mark.NAME1, pytest.mark.NAME2]

Hence, it can't detect custom defined attributes, and uses the special "pytestmark" keyword to do this.

It'd be neat to make a plugin.

AbdealiJK added a comment.EditedMay 19 2016, 6:47 AM

I've made https://github.com/AbdealiJK/pytest-attrib which should be good for at least our use case.
You can review it there and make issues at https://github.com/AbdealiJK/pytest-attrib/issues

(Old comment not saved when I unplugged the wrong thing.., which is now a bit redundant)
Regarding the mark decorator, I suspect that they could be used to achieve the same result, but the hard part will be building a solution that does not litter the test files with decorators needed only for py.test . Currently the test files are almost pure unittest`.

A easy way into this problem is to create a conftest.py which accepts a command line argument, like --site-only=en.wikipedia which then plays with the number of tests that are collected and executed based on the custom attributes.
Firstly, that is a mergable solution. And I suspect that by the time that has been built, the 'right' solution for a customisable nose-compatible plugin is then obvious.

I've made https://github.com/AbdealiJK/pytest-attrib which should be good for at least our use case.
You can review it there and make issues at https://github.com/AbdealiJK/pytest-attrib/issues

Awesome. Any chance you can add SITE_ONLY=1 support for py.test in .travis.yml?

I was hoping someone could review that and then I'd make the pip package for pytest-attrib.
I guess I'll make a beta pypi package and use that in the pywikibot travis and make another release as needed if there are issues.

Change 289605 had a related patch set uploaded (by AbdealiJK):
travis: Use pytest-attrib for SITE_ONLY=1 jobs

https://gerrit.wikimedia.org/r/289605

I ran the tests using pytest-attrib as mentioned in the gerrit comments, and found the number of tests being run on the nosetests and pytest were different, where the pytest builds always found more tests than the nose one.

PyTest build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131373171
Nose build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131404927

To be able to compare which tests were being not collected, I used the nose-ignore-docstring plugin to give the name of the function rather than the docstring.

I wrote the script http://pastebin.com/FnMQThYK which uses regex to find all the tests that were collected and run in the pytest and nose travis logs. When doing this, I found a lot of tests which inherit from DefaultSiteTestCase but were not being run by the SITE_ONLY=1 jobs in nose. This seems like a bug in nose, as the language and code attributes are set correctly, and only some of the subclasses of DefaultSiteTestCase are not collected, a lot of them are collected correctly.

Examples of subclasses of DefaultSiteTestCase which are collected by pytest but not by nose:

  • pagegenerators_tests:TestDayPageGenerator
  • pagegenerators_tests:LiveRCPageGeneratorTestCase
  • pagegenerators_tests:PageGeneratorIntersectTestCase
  • pagegenerators_tests:TestDequePreloadingGenerator
  • pagegenerators_tests:TestFactoryGenerator
  • pagegenerators_tests:TestLogeventsFactoryGenerator
  • patrolbot_tests:TestPatrolBot

Other than this, there were also some tests that were collected by nosetest, but not by pytest:

  • site_tests:TestPagePreloading
  • script_tests:TestScriptSimulate (This is only skipped in Job 18, but is found correctly in other jobs)
  • site_tests:TestSiteGenerators.test_allpages_langlinks_enabled

I've created appropriate issues at https://github.com/AbdealiJK/pytest-attrib/issues/2 and https://github.com/AbdealiJK/pytest-attrib/issues/1

On pytest-attrib the bugs were solved by using pytest for the parent class finding (which was buggy in python3).

The new pytest build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131373171
The new nosetests build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131404927

These were checked using the pastebin script mentioned above and I found that pytest-attrib always gets more tests than nose-attrib. Hence, this is now fixed upstream.

Change 289605 merged by jenkins-bot:
travis: Use pytest-attrib for SITE_ONLY=1 jobs

https://gerrit.wikimedia.org/r/289605

jayvdb closed this task as Resolved.May 20 2016, 1:31 PM