test_arabeyes is failing with assertion error
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dalba
	Mar 12 2017, 1:55 AM

Description

#3905 errored

https://travis-ci.org/wikimedia/pywikibot-core/jobs/210150734#L3943

_________________ NonStandardVersionSiteTestCase.test_arabeyes _________________

self = <tests.site_detect_tests.NonStandardVersionSiteTestCase testMethod=test_arabeyes>

    def test_arabeyes(self):

        """Test detection of MediaWiki sites for wiki.arabeyes.org."""

>       self.assertSite('http://wiki.arabeyes.org/$1')

tests/site_detect_tests.py:135: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.site_detect_tests.NonStandardVersionSiteTestCase testMethod=test_arabeyes>

url = 'http://wiki.arabeyes.org/$1'

    def assertSite(self, url):

        """

            Assert a MediaWiki site can be loaded from the url.

    

            @param url: Url of tested site

            @type url: str

            @raises AssertionError: Site under url is not MediaWiki powered

            """

        try:

            self.assertIsInstance(MWSite(url), MWSite)

        except (AttributeError, ConnectionError, RuntimeError, ServerError,

                Timeout):

>           raise AssertionError

E           AssertionError

tests/site_detect_tests.py:68: AssertionError

From 21 builds, 13 of them have this error and 8 of them have not run this test, i.e. the test has failed in all its runs.

The same thing has happened in 3904, 3903 and 3902.

There has been no error in 3901.

Details

	Subject	Repo	Branch	Lines +/-
	site_detect_tests.py: Skip tests on ServerError and Timeout	pywikibot/core	master	+2 -32

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Magul	T151145 Make Travis (and Appveyor) pass again
		Resolved		Dalba	T160268 test_arabeyes is failing with assertion error

Event Timeline

Dalba created this task.Mar 12 2017, 1:55 AM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMar 12 2017, 1:55 AM

Dalba renamed this task from test_arabeyes has failed with assertion error to test_arabeyes is failing with assertion error.Mar 12 2017, 1:56 AM

Dalba added a parent task: T151145: Make Travis (and Appveyor) pass again.

wiki.arabeyes.org server is down and the error is raised as AssertionError in assertSite.

I think the @_retry_few_times logic introduced in 917c7e115e9a8749386ada159a896af19418b3fe will be ineffective here. I'm gonna propose a change to skip site tests on server error.

Change 342319 had a related patch set uploaded (by Dalba):
[pywikibot/core] site_detect_tests.py: Skip tests on ServerError and Timeout

https://gerrit.wikimedia.org/r/342319

gerritbot added a project: Patch-For-Review.Mar 12 2017, 3:17 AM

Also test_wikichristian still fails every now and then, for example in the latest build:

https://travis-ci.org/wikimedia/pywikibot-core/jobs/210222916#L3960

The patch above should resolve that, too.

Change 342319 merged by jenkins-bot:
[pywikibot/core] site_detect_tests.py: Skip tests on ServerError and Timeout

https://gerrit.wikimedia.org/r/342319

I'm not sure completely removing @_retry_few_times is the way forwards. That was introduced to deal with a variety of hiccups when connecting to the sites. Yes if the site is down all will fail but that is less weird than having some builds fail and some not due to some hiccup.

Would it be possible to start with a ping and only run the test if that gets a positive reply? Alternatively we could reconsider running tests on live sites outside of our control.

Lokal_Profil added a subscriber: Magul.Mar 12 2017, 1:01 PM

Yes if the site is down all will fail but that is less weird than having some builds fail and some not due to some hiccup.

May be weird, but I think it's more convenient. When a test like this fails because of a server error, I have to go through at least 13 build logs to see what has happened. This is too much work, I think many won't bother to check them all and may miss other errors.

Now let's assume there is a hiccup, i.e. a test has been skipped and we have not noticed an error in a recent commit because the build has been successful.
It is very likely that the same test in one of the other 13 build jobs will fail, or at least will fail in the very next few builds.
Finding why test has not failed during the previous build is easy because those will be marked as skipped with a summary of why the have been skipped.
I think it will still be easy to track the cause.

Would it be possible to start with a ping and only run the test if that gets a positive reply?

I'm a little sceptical on that. It could help, maybe, but I'm not convinced that it's worth it.

Alternatively we could reconsider running tests on live sites outside of our control.

We could also store successful responses in an optional submodule and use that to run tests, but it seems to require much more effort.

Dalba removed a project: Patch-For-Review.Mar 12 2017, 2:56 PM

I'm going to mark this task as resolved. Please feel free to reopen or create a new task/RFC if required.

test_arabeyes is failing with assertion errorClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

test_arabeyes is failing with assertion error
Closed, ResolvedPublic
Actions

Related Objects
Search...