Page MenuHomePhabricator

Find an advanced test runner with logging
Open, HighPublic

Description

Currently the Jenkins tests and some of the Pywikibot travis jobs use nose, which includes some logging, but doesnt include timestamps. Timestamps are especially useful for the server side team when tracking down problems (e.g. T114824).

However many of the Travis-CI jobs only use unittest/unittest2, and one of the main advantages is that nose doesnt show the test method name if the test method includes a docstring. This makes it more difficult to find the test method, which is useful if attempting to re-run a specific test method, especially if it fails.

https://gerrit.wikimedia.org/r/#/c/244951/ proposed to use nose for almost all Travis jobs, however it was voted down (-1'd) because of this limitation.

There are several nose plugins that provide test method names:

nose2 may also be a way forward, except for T115296: nose2 fails to load Pywikibot tests

green also looks good, except for breakage described in #93, a few other 'very nice to have' open issues such as # 87, and an option to show the test module names has been requested: # 94.

Event Timeline

jayvdb created this task.Oct 13 2015, 9:40 AM
jayvdb claimed this task.
jayvdb raised the priority of this task from to High.
jayvdb updated the task description. (Show Details)
jayvdb added a project: Pywikibot-tests.
jayvdb added subscribers: jayvdb, XZise.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptOct 13 2015, 9:40 AM
jayvdb updated the task description. (Show Details)Oct 13 2015, 9:42 AM
jayvdb set Security to None.
jayvdb updated the task description. (Show Details)Oct 13 2015, 9:44 AM
jayvdb updated the task description. (Show Details)Oct 13 2015, 10:13 AM
jayvdb removed a subscriber: zhuyifei1999.
jayvdb added a subscriber: zhuyifei1999.

green doesnt yet include method names in the normal output, however on failure it does include them.

...
Skipped tests.script_tests.TestScriptSimulate.test__login - Skipping execution of auto-run scripts (set PYWIKIBOT2_TEST_AUTORUN=1 to enable) "login"

Captured stdout for tests.script_tests.TestScriptSimulate.test_editarticle
 killed  
Captured stdout for tests.script_tests.TestScriptSimulate.test_freebasemappingupload
 killed  
Captured stdout for tests.script_tests.TestScriptSimulate.test_harvest_template
 killed  
Captured stdout for tests.script_tests.TestScriptSimulate.test_imageuncat
 killed  
Captured stdout for tests.script_tests.TestScriptSimulate.test_interwiki
 10.522s 
Captured stdout for tests.script_tests.TestScriptSimulate.test_panoramiopicker
 10.478s 
Failure in tests.script_tests.TestScriptHelp.test_panoramiopicker
  File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
    testMethod()
  File "tests/script_tests.py", line 291, in testScript
    stderr_other)
  File "/usr/lib64/python2.7/unittest/case.py", line 843, in assertIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib64/python2.7/unittest/case.py", line 450, in fail
    raise self.failureException(msg)
AssertionError: u'Use -help for further information.' not found in [u'Traceback (most recent call last):', u'  File "<string>", line 1, in <module>', u'  File "pwb.py", line 242, in main', u'    run_python_file(filename, [filename] + args, argvu, file_package)', u'  File "pwb.py", line 120, in run_python_file', u'    main_mod.__dict__)', u'  File "./scripts/panoramiopicker.py", line 65, in <module>', u"    site=pywikibot.Site(u'commons', u'commons')):", u"TypeError: 'NoneType' object is not callable", u'CRITICAL: Closing network session.', u'']

Ran 125 tests in 464.448s

FAILED (expected_failures=1, failures=1, passes=122, skips=1)
jayvdb updated the task description. (Show Details)Oct 13 2015, 10:17 AM
jayvdb updated the task description. (Show Details)Oct 13 2015, 11:37 AM

Looking a bit more broadly, at other unittest compatible test runners:

https://pypi.python.org/pypi/testify and https://pypi.python.org/pypi/haas look like they may have potential, but both command line tools failed to load the Pywikibot test suite for me, and/or failed to load specific modules.

$ haas tests.site_tests

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

haas shows the timestamps for each test

[pep8]$ haas -v
[Tue Oct 13 23:38:28 2015] ( 1/30) test_checkers_testsuite (testsuite.test_all.Pep8TestCase) ... ok
[Tue Oct 13 23:38:29 2015] ( 2/30) test_doctest (testsuite.test_all.Pep8TestCase) ... ok
[Tue Oct 13 23:38:30 2015] ( 3/30) test_own_dog_food (testsuite.test_all.Pep8TestCase) ... ok
[Tue Oct 13 23:38:31 2015] ( 4/30) test_selftest (testsuite.test_all.Pep8TestCase) ... ok
[Tue Oct 13 23:38:31 2015] ( 5/30) test_check_nullbytes (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] ( 6/30) test_check_unicode (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] ( 7/30) test_register_ast_check (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] ( 8/30) test_register_invalid_check (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] ( 9/30) test_register_logical_check (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] (10/30) test_register_physical_check (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] (11/30) test_styleguide (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] (12/30) test_styleguide_check_files (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] (13/30) test_styleguide_checks (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] (14/30) test_styleguide_continuation_line_outdented (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] (15/30) test_styleguide_excluded (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:31 2015] (16/30) test_styleguide_ignore_code (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:32 2015] (17/30) test_styleguide_init_report (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:32 2015] (18/30) test_styleguide_options (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:32 2015] (19/30) test_styleguide_unmatched_triple_quotes (testsuite.test_api.APITestCase) ... ok
[Tue Oct 13 23:38:32 2015] (20/30) test_normalize_paths (testsuite.test_util.UtilTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (21/30) test_check_diff (testsuite.test_shell.ShellTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (22/30) test_check_noarg (testsuite.test_shell.ShellTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (23/30) test_check_non_existent (testsuite.test_shell.ShellTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (24/30) test_check_simple (testsuite.test_shell.ShellTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (25/30) test_check_stdin (testsuite.test_shell.ShellTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (26/30) test_print_usage (testsuite.test_shell.ShellTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (27/30) test_multiline_ignore_parsing (testsuite.test_parser.ParserTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (28/30) test_multiline_trailing_comma_ignore_parsing (testsuite.test_parser.ParserTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (29/30) test_trailing_comma_ignore_parsing (testsuite.test_parser.ParserTestCase) ... ok
[Tue Oct 13 23:38:32 2015] (30/30) test_vanilla_ignore_parsing (testsuite.test_parser.ParserTestCase) ... ok

https://pypi.python.org/pypi/falafel sounds good, but doesnt include a command line tool.

Twisted's trial version 15.1.0 fails very badly, but it might be easily fixed as the problem seems to be always the same error.

tests.api_tests.TestPropertyGenerator.test_info
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
    testMethod()
  File "/home/jayvdb/projects/pywiki/core/tests/api_tests.py", line 682, in test_one_continuation
    mainpage = self.get_mainpage()
  File "/home/jayvdb/projects/pywiki/core/tests/aspects.py", line 997, in get_mainpage
    site = self.get_site()
  File "/home/jayvdb/projects/pywiki/core/tests/aspects.py", line 985, in <lambda>
    self.get_site = lambda name=None: self.__class__.get_site(name)
  File "/home/jayvdb/projects/pywiki/core/tests/aspects.py", line 963, in get_site
    return cls.sites[name]['site']
exceptions.KeyError: u'site'

Also, pytest may be another alternative if you're looking to replace nose.
I've used it before and it was really good. It's customizable with hooks too

jayvdb added a comment.May 5 2016, 3:10 PM

Also, pytest may be another alternative if you're looking to replace nose.
I've used it before and it was really good. It's customizable with hooks too

Strange I didnt mention it. IIRC it had problems with our voodoo. Probably our metaclass parameterisation of tests - see date_tests and archivebot_tests.

I tried out pytest. The complete output can be found here: https://travis-ci.org/AbdealiJK/pywikibot-core/builds/128659687
Final stats found (for the 1st travis build) was: 10 failed, 1787 passed, 282 skipped, 40 xfailed, 5 pytest-warnings, 1 error in 2135.09 seconds

I think this is actually pretty good, I havent checked out all the failed tests, but a few were because of TimeOutErrors and a few just failed (AssertionErrors, Not sure why). I don't have a complete understanding of all the tests, so maybe someone more experienced could take a look at the logs.

I think it may have something to do with me not having a sysops (Some failed messages had "no sysop configured" in them) ?

@jayvdb About the metaclass parametrization you mentioned, it worked fine. date_tests and archivebot_tests worked as expected.

Also, It'd be a good idea to make a simple checklist of all the things that we need in a test system. I'm not sure what is required in a test suite.

Pytest's output is like this:

tests/add_text_tests.py ..
tests/api_tests.py ........................................s..s....F.s.....
## ... etc etc etc ... ##

But it can be customized using pytest-sugar and also using the pytest hooks in conftest

I checked out the 10 tests that failed in the above mentioned comment. Some failed because of a ConnectionError or a timeout of some sort. Except 1 test, those that got an AssertionError, fail with nosetests too, so it's not pytest's fault.

The only exception was tests.wikistats_tests.WikiStatsTestCase's text_xml and test_sort. These tests don't fail locally on my system, but fail on Travis. So, again probably not pytest's fault, but I am not sure about this.

So, I think if we can sort out the kind of output that's needed for the testing infrastructure, pytest could be a possibility.

The main requirements
*logging support (like nose, shows the stdout, stderr, & logging on failure)

  • re-run-ability -- it should be easily to identify a failed test, and then run that specific test (see examples in tests/README.rst showing how to run a specific test)
  • timestamps -- a job can take an hour, but we need more specific times for errors, in order to use when looking at the (very large) server logs.
  • ability to block network access (currently Jenkins uses a nose plugin for this)

Change 289385 had a related patch set uploaded (by John Vandenberg):
Use py.test for some Travis jobs with 300s timeout

https://gerrit.wikimedia.org/r/289385

Change 289385 merged by jenkins-bot:
Use py.test for some Travis jobs with 300s timeout

https://gerrit.wikimedia.org/r/289385

Need to confirm that PYWIKIBOT_TEST_MODULES works with py.test / nose , and find an alternative approach if it doesnt work.

We can make a conftest to respect the collection of the PYWIKIBOT_TEST_MODULES in pytest

Dalba moved this task from Backlog to Framework on the Pywikibot-tests board.May 5 2017, 4:36 AM