Lot of AssertionError: "^$" does not match "None" on Travis CI
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Magul
	Feb 19 2017, 10:26 PM

Description

For example:

_____________________ LiveBotTestCase.test_ExistingPageBot _____________________
self = <tests.bot_tests.LiveBotTestCase testMethod=test_ExistingPageBot>
    def test_ExistingPageBot(self):
        """Test ExistingPageBot class."""
        def post_treat(page):
            """Verify the page exists."""
            self.assertTrue(page.exists())
    
        self._treat_site = None
        self.bot = pywikibot.bot.ExistingPageBot(
            generator=self._missing_generator())
        self.bot.treat_page = self._treat_page(post_treat=post_treat)
        self.bot.exit = self._exit()
>       self.bot.run()
tests/bot_tests.py:345: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pywikibot/bot.py:1453: in run
    self.exit()
tests/bot_tests.py:331: in exit
    super(LiveBotTestCase, self)._exit(t, written, exception)()
tests/bot_tests.py:174: in exit
    self.assertRaisesRegex(StopIteration, '^$', next, self._page_iter)
E   AssertionError: "^$" does not match "None"

For more see: https://travis-ci.org/wikimedia/pywikibot-core/jobs/203198988

Details

	Subject	Repo	Branch	Lines +/-
	[test] User assertRaisesRegex again	pywikibot/core	master	+2 -2
	bot_tests.py: Use assertRaises for StopIteration('')	pywikibot/core	master	+2 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Magul	T151145 Make Travis (and Appveyor) pass again
Invalid	None	T72987 Python 2.6 issues (tracking)
Resolved	Dalba	T158519 Lot of AssertionError: "^$" does not match "None" on Travis CI

Event Timeline

Magul created this task.Feb 19 2017, 10:26 PM

I've checked few dozens of builds on our Travis and it looks, that this one first occurs in here. The commit that introduced it was fix for Py2.6 support after introduction of pytest-httpbin.

CC: @Phantom42 @jayvdb

This particular line that is failing has been introduced in our code in rPWBC2c33f951

I've checked on Py2,6 and Py2.7 and str(StopIteration()) results with '' so this assertion should be meet in either way. I also checked, that we use unittest2 as unittest for Python versions lower then 2.7.3. I believe problem could be somwhere there.

I am currently trying to undestand what is the reason for that error.

Actually, I am not sure that d6a8ea397f6a introduced the error.
After 431e72083ac1 there were some problems with travis test builds because of unsupported syntax for python 2.6 was being used (tests were not even executed).
And d6a8ea397f6a fixed it. So any commit between 431e72083ac1 and d6a8ea397f6a could introduce that error.

Well, after some git bisect-ing and experimenting with code I understood, that 431e72083ac1 and d6a8ea397f6a are unrelated to this problem. I checked out some commits from december 2016 (we were not using pytest-httpbin at that point). As expected, tests were not failing. Then I changed line 174 of bot_tests.py to use assertRaisesRegex instead of assertRaises as it was done in rPWBC2c33f951d7b0. Tests began to fail again. That means that rPWBC2c33f951d7b0 is the reason for tests failing. And we didn't notice that problem until we fixed some syntax issues in d6a8ea397f6a, which prevented tests from executing.

In my opinon, something is wrong with regular expression used. So I think we just need to change it to something else to fix the problem. @Magul what are your thoughts on this? Do you agree?

@Phantom42 Yes, I agree. Actually if You would read my comments above, I done the same research few days ago.

I'm certain that this is an issue with this assertRaiseRegex but I choose to take tougher path and to find actual root cause.

I found that for Py2.6 we use unittest2 as fallback for unittest but i I use Py2.7 with unittests2 I the issue doesn't occur.

I will look at it further.

Dalba added a parent task: T72987: Python 2.6 issues (tracking).Mar 15 2017, 12:39 AM

python2.6 passes exception arguments to a context manager differently. Here is an example script:

i = iter([])

class PrintExitArgs:
    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print(exc_type, exc_value, traceback)
        return True

with PrintExitArgs():
    next(i)

here are the results of running this script on python2.6 and 2.7:

$ python2.6 myscript.py
(<type 'exceptions.StopIteration'>, None, <traceback object at 0x7fa9349a9830>)
$ python2.7 myscript.py
(<type 'exceptions.StopIteration'>, StopIteration(), <traceback object at 0x7f38f0ab3998>)

The _AssertRaisesContext in unittest2 has the same issue. The exc_value it receives is None.

I did not find their official bug tracker to report the issue.

AFAIS we can't use assertRaisesRegex in this particular case until the issue is resolved in the unittest2. Until then I'm going to propose a change to use assertRaises instead.

Change 342830 had a related patch set uploaded (by Dalba):
[pywikibot/core] bot_tests.py: Use assertRaises for StopIteration('')

https://gerrit.wikimedia.org/r/342830

gerritbot added a project: Patch-For-Review.Mar 15 2017, 1:39 PM

Change 342830 merged by jenkins-bot:
[pywikibot/core] bot_tests.py: Use assertRaises for StopIteration('')

https://gerrit.wikimedia.org/r/342830

Dalba closed this task as Resolved.Mar 16 2017, 9:56 AM

Change 435716 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [test] User assertRaisesRegex again

https://gerrit.wikimedia.org/r/435716

Change 435716 merged by jenkins-bot:
[pywikibot/core@master] [test] User assertRaisesRegex again