Page MenuHomePhabricator

pytest doesnt respect __test__ = False on script_tests metaclass created methods
Closed, ResolvedPublic


When pytest is used on the tests, it fails on scripts_tests test_version (help & simulate), where it should have skipped them.

According to it supports __test__ attribute on modules/classes/functions . It doesnt mention supporting methods.

Event Timeline

jayvdb created this task.May 18 2016, 8:25 AM
Restricted Application added subscribers: pywikibot-bugs-list, Zppix, Aklapper. · View Herald TranscriptMay 18 2016, 8:25 AM
AbdealiJK added a comment.EditedMay 18 2016, 3:15 PM

I don't think the problem is the __test__. According to the code in script_tests unrunnable_script_list has two files that set __test__ - version and script_wui.

Without dct[test_name] = unittest.expectedFailure(dct[test_name]), script_wui is skipped because of xfail:

$ python -m pytest -vvv -k "TestScriptHelp and test_script_wui or TestScriptHelp and test_version"

collected 2146 items 

tests/ <- ../../../../../usr/lib/python2.7/unittest/ xfail
tests/ FAILED

But they are not supposed to be run at all, if __test__ is False.

script_wui is listed in lots of "failure lists" in that test module, including the script_deps, and no_args_expected_results, so it is not surprising that it shows up as xfail for some other reason.

AbdealiJK added a comment.EditedMay 18 2016, 4:26 PM

Yeah, You're right. It should not even be collected.

I also made this simpler testcase to test this out:

import unittest

def test_abc():

def test_abc_not_run():

test_abc_not_run.__test__ = False

class ClassTest(unittest.TestCase):
    def test_class_abc(self):

class NotRunClassTest(unittest.TestCase):
    __test__ = False

    def test_class_abc_not_run(self):

class ClassMeta(type):

    def __new__(cls, name, bases, dct):
        for name in ['abc', 'abc_not_run']:
            test_name = 'test_metaclass_' + name
            dct[test_name] = lambda self: None
            dct[test_name].__name__ = test_name

        dct['test_metaclass_abc_not_run'].__test__ = False
        return super(ClassMeta, cls).__new__(cls, name, bases, dct)

class ClassMetaTest(unittest.TestCase):
    __metaclass__ = ClassMeta

And I used the conftest:

def pytest_runtest_setup(item):
    print '    __test__ for', item, "is", getattr(item._obj, '__test__', None)

Results got:

$ py.test --capture=no -vvv
platform linux2 -- Python 2.7.6, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 -- /home/ajk/Documents/wiki/venv/bin/python
collected 4 items 
    __test__ for <TestCaseFunction 'test_metaclass_abc'> is None
    __test__ for <TestCaseFunction 'test_metaclass_abc_not_run'> is False
    __test__ for <Function 'test_abc'> is None
    __test__ for <TestCaseFunction 'test_class_abc'> is None

===================================== 4 passed in 2.37 seconds =====================================

Created upstream.

jayvdb renamed this task from pytest doesnt respect __test__ = False on script_tests metaclass to pytest doesnt respect __test__ = False on script_tests metaclass created methods.May 18 2016, 5:21 PM
jayvdb added a project: Upstream.
jayvdb moved this task from Backlog to Reported Upstream on the Upstream board.
jayvdb updated the task description. (Show Details)

This has been merged upstream. They mentioned that they would be doing a release in a week or so.

PR is at:

jayvdb moved this task from Backlog to Upstream issues on the Pywikibot board.May 19 2016, 2:20 PM
AbdealiJK closed this task as Resolved.Sep 4 2016, 6:23 AM

This has been merged into the stable pytest 3.0
Closing the task