When pytest is used on the tests, it fails on scripts_tests test_version (help & simulate), where it should have skipped them.
According to https://pytest.org/latest/nose.html it supports __test__ attribute on modules/classes/functions . It doesnt mention supporting methods.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T152961 Improve tests suite | |||
| Open | None | T129368 Usages of unittest.expectedFailure or unittest.skip should be solved (goal) | |||
| Declined | None | T115313 Find an advanced test runner with logging | |||
| Resolved | AbdealiJK | T135594 pytest doesnt respect __test__ = False on script_tests metaclass created methods |
I don't think the problem is the __test__. According to the code in script_tests unrunnable_script_list has two files that set __test__ - version and script_wui.
Without dct[test_name] = unittest.expectedFailure(dct[test_name]), script_wui is skipped because of xfail:
$ python -m pytest -vvv -k "TestScriptHelp and test_script_wui or TestScriptHelp and test_version" # REMOVED CONTENT # collected 2146 items tests/script_tests.py::TestScriptHelp::test_script_wui <- ../../../../../usr/lib/python2.7/unittest/case.py xfail tests/script_tests.py::TestScriptHelp::test_version FAILED
But they are not supposed to be run at all, if __test__ is False.
script_wui is listed in lots of "failure lists" in that test module, including the script_deps, and no_args_expected_results, so it is not surprising that it shows up as xfail for some other reason.
Yeah, You're right. It should not even be collected.
I also made this simpler testcase to test this out:
import unittest
def test_abc():
pass
def test_abc_not_run():
pass
test_abc_not_run.__test__ = False
class ClassTest(unittest.TestCase):
def test_class_abc(self):
pass
class NotRunClassTest(unittest.TestCase):
__test__ = False
def test_class_abc_not_run(self):
pass
class ClassMeta(type):
def __new__(cls, name, bases, dct):
for name in ['abc', 'abc_not_run']:
test_name = 'test_metaclass_' + name
dct[test_name] = lambda self: None
dct[test_name].__name__ = test_name
dct['test_metaclass_abc_not_run'].__test__ = False
return super(ClassMeta, cls).__new__(cls, name, bases, dct)
class ClassMetaTest(unittest.TestCase):
__metaclass__ = ClassMetaAnd I used the conftest:
def pytest_runtest_setup(item):
print
print ' __test__ for', item, "is", getattr(item._obj, '__test__', None)Results got:
$ py.test --capture=no -vvv platform linux2 -- Python 2.7.6, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 -- /home/ajk/Documents/wiki/venv/bin/python #REMOVED CONTENT# collected 4 items test_abc.py::ClassMetaTest::test_metaclass_abc __test__ for <TestCaseFunction 'test_metaclass_abc'> is None PASSED test_abc.py::ClassMetaTest::test_metaclass_abc_not_run __test__ for <TestCaseFunction 'test_metaclass_abc_not_run'> is False PASSED test_abc.py::test_abc __test__ for <Function 'test_abc'> is None PASSED test_abc.py::ClassTest::test_class_abc __test__ for <TestCaseFunction 'test_class_abc'> is None PASSED ===================================== 4 passed in 2.37 seconds =====================================
Created https://github.com/pytest-dev/pytest/issues/1558 upstream.
This has been merged upstream. They mentioned that they would be doing a release in a week or so.