Page MenuHomePhabricator

pytest doesnt respect __test__ = False on script_tests metaclass created methods
Closed, ResolvedPublic

Description

When pytest is used on the tests, it fails on scripts_tests test_version (help & simulate), where it should have skipped them.

According to https://pytest.org/latest/nose.html it supports __test__ attribute on modules/classes/functions . It doesnt mention supporting methods.

Event Timeline

jayvdb created this task.May 18 2016, 8:25 AM
Restricted Application added subscribers: pywikibot-bugs-list, Zppix, Aklapper. · View Herald TranscriptMay 18 2016, 8:25 AM
AbdealiJK added a comment.EditedMay 18 2016, 3:15 PM

I don't think the problem is the __test__. According to the code in script_tests unrunnable_script_list has two files that set __test__ - version and script_wui.

Without dct[test_name] = unittest.expectedFailure(dct[test_name]), script_wui is skipped because of xfail:

$ python -m pytest -vvv -k "TestScriptHelp and test_script_wui or TestScriptHelp and test_version"

# REMOVED CONTENT #
collected 2146 items 

tests/script_tests.py::TestScriptHelp::test_script_wui <- ../../../../../usr/lib/python2.7/unittest/case.py xfail
tests/script_tests.py::TestScriptHelp::test_version FAILED

But they are not supposed to be run at all, if __test__ is False.

script_wui is listed in lots of "failure lists" in that test module, including the script_deps, and no_args_expected_results, so it is not surprising that it shows up as xfail for some other reason.

AbdealiJK added a comment.EditedMay 18 2016, 4:26 PM

Yeah, You're right. It should not even be collected.

I also made this simpler testcase to test this out:

import unittest

def test_abc():
    pass

def test_abc_not_run():
    pass

test_abc_not_run.__test__ = False

class ClassTest(unittest.TestCase):
    def test_class_abc(self):
        pass

class NotRunClassTest(unittest.TestCase):
    __test__ = False

    def test_class_abc_not_run(self):
        pass


class ClassMeta(type):

    def __new__(cls, name, bases, dct):
        for name in ['abc', 'abc_not_run']:
            test_name = 'test_metaclass_' + name
            dct[test_name] = lambda self: None
            dct[test_name].__name__ = test_name

        dct['test_metaclass_abc_not_run'].__test__ = False
        return super(ClassMeta, cls).__new__(cls, name, bases, dct)

class ClassMetaTest(unittest.TestCase):
    __metaclass__ = ClassMeta

And I used the conftest:

def pytest_runtest_setup(item):
    print
    print '    __test__ for', item, "is", getattr(item._obj, '__test__', None)

Results got:

$ py.test --capture=no -vvv
platform linux2 -- Python 2.7.6, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 -- /home/ajk/Documents/wiki/venv/bin/python
#REMOVED CONTENT#
collected 4 items 

test_abc.py::ClassMetaTest::test_metaclass_abc 
    __test__ for <TestCaseFunction 'test_metaclass_abc'> is None
PASSED
test_abc.py::ClassMetaTest::test_metaclass_abc_not_run 
    __test__ for <TestCaseFunction 'test_metaclass_abc_not_run'> is False
PASSED
test_abc.py::test_abc 
    __test__ for <Function 'test_abc'> is None
PASSED
test_abc.py::ClassTest::test_class_abc 
    __test__ for <TestCaseFunction 'test_class_abc'> is None
PASSED

===================================== 4 passed in 2.37 seconds =====================================

Created https://github.com/pytest-dev/pytest/issues/1558 upstream.

jayvdb renamed this task from pytest doesnt respect __test__ = False on script_tests metaclass to pytest doesnt respect __test__ = False on script_tests metaclass created methods.May 18 2016, 5:21 PM
jayvdb added a project: Upstream.
jayvdb moved this task from Backlog to Reported Upstream on the Upstream board.
jayvdb updated the task description. (Show Details)

This has been merged upstream. They mentioned that they would be doing a release in a week or so.

PR is at: https://github.com/pytest-dev/pytest/pull/1561

jayvdb moved this task from Backlog to Upstream issues on the Pywikibot board.May 19 2016, 2:20 PM
AbdealiJK moved this task from Backlog to Upstream on the Pywikibot-tests board.May 20 2016, 1:44 AM
AbdealiJK closed this task as Resolved.Sep 4 2016, 6:23 AM

This has been merged into the stable pytest 3.0
Closing the task