Page MenuHomePhabricator

TestFactoryGenerator.test_pages_with_property_generator is failing with TimeoutError on wikidata
Closed, ResolvedPublic

Description

___________ TestFactoryGenerator.test_pages_with_property_generator ____________

self = <tests.pagegenerators_tests.TestFactoryGenerator testMethod=test_pages_with_property_generator>

    def test_pages_with_property_generator(self):

        """Test the pages_with_property_generator method."""

        mysite = self.get_site()

        for item in ('defaultsort', 'disambiguation', 'displaytitle',

                     'hiddencat', 'invalid_property'):

>           if item in mysite.get_property_names():

tests/pagegenerators_tests.py:1026: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

pywikibot/site.py:1357: in callee

    return fn(self, *args, **kwargs)

pywikibot/site.py:6652: in get_property_names

    self._property_names = [pn['propname'] for pn in ppngen]

pywikibot/data/api.py:2753: in __iter__

    self.data = self.request.submit()

pywikibot/data/api.py:1978: in submit

    self.wait()

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = pywikibot.data.api.Request<wikidata:wikidata->'/w/api.php?maxlag=5&continue=&f...agepropnames&meta=userinfo&indexpageids=&action=query&uiprop=blockinfo|hasmsg'>

    def wait(self):

        """Determine how long to wait after a failed request."""

        self.max_retries -= 1

        if self.max_retries < 0:

>           raise TimeoutError("Maximum retries attempted without success.")

E           TimeoutError: Maximum retries attempted without success.

pywikibot/data/api.py:2197: TimeoutError

_________________ TestSiteGenerators.test_pages_with_property __________________

self = <tests.site_tests.TestSiteGenerators testMethod=test_pages_with_property>

    def test_pages_with_property(self):

        """Test pages_with_property method."""

        if MediaWikiVersion(self.site.version()) < MediaWikiVersion('1.21'):

            raise unittest.SkipTest('requires v1.21+')

        mysite = self.get_site()

>       pnames = mysite.get_property_names()

tests/site_tests.py:1002: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

pywikibot/site.py:1357: in callee

    return fn(self, *args, **kwargs)

pywikibot/site.py:6652: in get_property_names

    self._property_names = [pn['propname'] for pn in ppngen]

pywikibot/data/api.py:2753: in __iter__

    self.data = self.request.submit()

pywikibot/data/api.py:2342: in submit

    self._data = super(CachedRequest, self).submit()

pywikibot/data/api.py:1978: in submit

    self.wait()

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = tests.TestRequest<wikidata:wikidata->'/w/api.php?maxlag=5&continue=&format=jso...agepropnames&meta=userinfo&indexpageids=&action=query&uiprop=blockinfo|hasmsg'>

    def wait(self):

        """Determine how long to wait after a failed request."""

        self.max_retries -= 1

        if self.max_retries < 0:

>           raise TimeoutError("Maximum retries attempted without success.")

E           TimeoutError: Maximum retries attempted without success.
build[.job]test_pages_with_property_generatortest_pages_with_property
4000FAILEDFAILED
3999.21FAILEDFAILED
3998.21FAILEDFAILED
3997passedpassed
3996.21FAILEDFAILED
3995.21FAILEDFAILED
3994passedpassed
3993passedpassed
3992.21FAILEDFAILED
3991.21FAILEDFAILED
3990.1 (enwp)passedFAILED
3990.3 (enwp)FAILEDpassed
3991.21FAILEDFAILED
3989passedpassed
3988passedpassed
3987passedpassed
3986passedpassed
3985passedpassed

Event Timeline

Change 349788 had a related patch set uploaded (by Xqt):
[pywikibot/core@master] Allow builds to retry twice

https://gerrit.wikimedia.org/r/349788

Xqt triaged this task as High priority.Apr 23 2017, 11:37 AM

As I said in T115825 the problem is the following query:

I tested the response time with the following script from Tool Labs:

from statistics import mean
from time import time

import requests

times = []
for i in range(10):
    t0 = time()
    requests.get(
        'https://www.wikidata.org/w/'
        'api.php?list=pagepropnames&action=query&indexpageids=&'
        'continue=&ppnlimit=500&meta=userinfo&'
        'uiprop=blockinfo%7Chasmsg&maxlag=15&format=json')
    td = time() - t0
    times.append(td)
    print(i, td)

print('mean:', mean(times))
print('max:', max(times))

The result:

0 50.27381682395935
1 47.168055057525635
2 43.26734662055969
3 50.04540014266968
4 45.82349920272827
5 47.00782895088196
6 46.657241106033325
7 45.37545084953308
8 44.3095428943634
9 40.617267370224

mean: 46.05454490184784
max: 50.27381682395935

According to these results, I don't think increasing the max_retries will help. (It's always over 30 seconds)

Increasing the read timeout should fix the problem, but we can't increase read timeout without increasing connect timeout, because the version of requests on wmflabs does not support that. See T103069. The current version of requests on labs is 2.2.1. Separating read and connect timeouts requires requests 2.4.0.

I propose we change the default socket_timeout to 75 seconds.

Dalba removed Dalba as the assignee of this task.Apr 23 2017, 11:55 AM

Change 349790 had a related patch set uploaded (by Dalba):
[pywikibot/core@master] config2.py: Increase the default timeout to 75 seconds

https://gerrit.wikimedia.org/r/349790

Change 349788 abandoned by Xqt:
Allow builds to retry twice

Reason:
Per Dalba's comment

https://gerrit.wikimedia.org/r/349788

Xqt claimed this task.

Change 349790 merged by jenkins-bot:
[pywikibot/core@master] config2.py: Increase the default socket_timeout to 75 seconds

https://gerrit.wikimedia.org/r/349790