Pywikibot uses nose attributes to select only tests that run on a certain site. This is used for the Travis-CI SITE_ONLY=1 jobs, so they run less unit tests on a specific host only in a short period of time.
py.test doesnt have any similar functionality. As it is a nose plugin, it should probably be re-implemented as a py.test plugin , and py.test says they merge any very useful plugins into the core product once it is stable.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| [cleanup] Travis tests never use nose anymore | pywikibot/core | master | +3 -11 | |
| travis: Use pytest-attrib for SITE_ONLY=1 jobs | pywikibot/core | master | +18 -13 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T115313 Find an advanced test runner with logging | |||
| Resolved | AbdealiJK | T135659 py.test doesnt support selecting tests using test object attributes |
pytest does have an equivalent pytest.mark, but it seems to be a little limited. It can only be used by doing:
Hence, it can't detect custom defined attributes, and uses the special "pytestmark" keyword to do this.
It'd be neat to make a plugin.
I've made https://github.com/AbdealiJK/pytest-attrib which should be good for at least our use case.
You can review it there and make issues at https://github.com/AbdealiJK/pytest-attrib/issues
(Old comment not saved when I unplugged the wrong thing.., which is now a bit redundant)
Regarding the mark decorator, I suspect that they could be used to achieve the same result, but the hard part will be building a solution that does not litter the test files with decorators needed only for py.test . Currently the test files are almost pure unittest`.
A easy way into this problem is to create a conftest.py which accepts a command line argument, like --site-only=en.wikipedia which then plays with the number of tests that are collected and executed based on the custom attributes.
Firstly, that is a mergable solution. And I suspect that by the time that has been built, the 'right' solution for a customisable nose-compatible plugin is then obvious.
I was hoping someone could review that and then I'd make the pip package for pytest-attrib.
I guess I'll make a beta pypi package and use that in the pywikibot travis and make another release as needed if there are issues.
Change 289605 had a related patch set uploaded (by AbdealiJK):
travis: Use pytest-attrib for SITE_ONLY=1 jobs
I ran the tests using pytest-attrib as mentioned in the gerrit comments, and found the number of tests being run on the nosetests and pytest were different, where the pytest builds always found more tests than the nose one.
PyTest build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131373171
Nose build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131404927
To be able to compare which tests were being not collected, I used the nose-ignore-docstring plugin to give the name of the function rather than the docstring.
I wrote the script http://pastebin.com/FnMQThYK which uses regex to find all the tests that were collected and run in the pytest and nose travis logs. When doing this, I found a lot of tests which inherit from DefaultSiteTestCase but were not being run by the SITE_ONLY=1 jobs in nose. This seems like a bug in nose, as the language and code attributes are set correctly, and only some of the subclasses of DefaultSiteTestCase are not collected, a lot of them are collected correctly.
Examples of subclasses of DefaultSiteTestCase which are collected by pytest but not by nose:
Other than this, there were also some tests that were collected by nosetest, but not by pytest:
I've created appropriate issues at https://github.com/AbdealiJK/pytest-attrib/issues/2 and https://github.com/AbdealiJK/pytest-attrib/issues/1
On pytest-attrib the bugs were solved by using pytest for the parent class finding (which was buggy in python3).
The new pytest build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131373171
The new nosetests build - https://travis-ci.org/AbdealiJK/pywikibot-core/builds/131404927
These were checked using the pastebin script mentioned above and I found that pytest-attrib always gets more tests than nose-attrib. Hence, this is now fixed upstream.
Change 665554 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [cleanup] Travis tests never use nose anymore
Change 665554 merged by jenkins-bot:
[pywikibot/core@master] [cleanup] Travis tests never use nose anymore