Page MenuHomePhabricator
Create Task
Maniphest T242083

test_page_from_repository fails with EntityTypeUnknownException on multiple sites
Closed, ResolvedPublic
Actions

Description

https://travis-ci.org/wikimedia/pywikibot/jobs/633377685#L1496
https://api.travis-ci.org/v3/job/679015138/log.txt

======================================================================

ERROR: test_item (tests.site_tests.TestDataSitePreloading)

Test that ItemPage preloading works for Item objects.

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/wikimedia/pywikibot/tests/site_tests.py", line 3358, in test_item

    for num in range(1, 6)]

  File "/home/travis/build/wikimedia/pywikibot/tests/site_tests.py", line 3358, in <listcomp>

    for num in range(1, 6)]

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/page.py", line 4416, in __init__

    ns = site.item_namespace

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/site.py", line 7668, in item_namespace

    self._item_namespace = self.get_namespace_for_entity_type('item')

  File "/home/travis/build/wikimedia/pywikibot/pywikibot/site.py", line 7657, in get_namespace_for_entity_type

    .format(self, entity_type))

pywikibot.exceptions.EntityTypeUnknownException: DataSite("wikidata", "wikidata") does not support entity type "item"
___________ TestCategoryFromWikibase.test_page_from_repository_de_wp ___________

self = <tests.site_tests.TestCategoryFromWikibase testMethod=test_page_from_repository_de_wp>

    def wrapped_method(self):

        sitedata = self.sites[key]

        self.site_key = key

        self.family = sitedata['family']

        self.code = sitedata['code']

        self.site = sitedata['site']

>       func(self, key)

tests/aspects.py:748: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

tests/site_tests.py:3787: in test_page_from_repository

    page = site.page_from_repository(self.ITEM)

pywikibot/site.py:2814: in page_from_repository

    dp = pywikibot.ItemPage(repo, item)

pywikibot/page.py:4422: in __init__

    ns = site.item_namespace

pywikibot/site.py:7668: in item_namespace

    self._item_namespace = self.get_namespace_for_entity_type('item')

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = DataSite("wikidata", "wikidata"), entity_type = 'item'

    def get_namespace_for_entity_type(self, entity_type):

        """

        Return namespace for given entity type.

    

        @return: corresponding namespace

        @rtype: Namespace

        """

        if not hasattr(self, '_entity_namespaces'):

            self._cache_entity_namespaces()

        if entity_type in self._entity_namespaces:

            return self._entity_namespaces[entity_type]

        raise EntityTypeUnknownException(

            '{0!r} does not support entity type "{1}"'

>           .format(self, entity_type))

E       EntityTypeUnknownException: DataSite("wikidata", "wikidata") does not support entity type "item"

pywikibot/site.py:7657: EntityTypeUnknownException

Related Objects
Search...

Event Timeline

Dvorapa created this task.Jan 7 2020, 9:48 AM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptJan 7 2020, 9:48 AM
Dvorapa merged a task: T243666: Travis fails on test_page_from_repository (TestPageFromWikibase, TestCategoryFromWikibase).Jan 25 2020, 12:14 PM
Dvorapa triaged this task as High priority.Jan 25 2020, 12:19 PM
Dvorapa updated the task description. (Show Details)Jan 28 2020, 9:49 AM
Xqt added a subscriber: matej_suchanek.Jan 28 2020, 10:38 AM
Dvorapa added a comment.EditedFeb 5 2020, 8:46 PM

Introduced with rPWBCa892b382d15253f2f66a9f88a563090bb647710e

Dvorapa added a subscriber: Xqt.EditedFeb 5 2020, 9:03 PM

@Xqt @matej_suchanek The error occurs only sometimes and it makes test produce milions of errors at once. I think this has something to do with T242081 as the 'item' is loaded into cache with a WD namespaces request. WD requests fail lately with timeout and thus can result in a broken cache. Which makes errors like this, because there is no cache-check or cache-repair mechanism or namespace-check or namespace-repair mechanism as well

Dvorapa added a comment.Feb 5 2020, 9:08 PM

But the code haven't changed much in the mentioned patch, which is weird

Dvorapa added a comment.Feb 5 2020, 10:23 PM

Okay, if DataSite.namespaces fails to get namespaces (due to timeout), this error is thrown, which is not a good behavior

Dvorapa added a comment.Feb 5 2020, 10:25 PM

So this is a duplicate of T242081, but the error message is not much useful

Dvorapa added a comment.Feb 5 2020, 10:33 PM

Basically, here (_build_namespaces): https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/site.py$2618 should be a check that all of these (builtin_namespaces): https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/site.py$406 are included, otherwise it should fail with API request error and not return an empty list of namespaces to continue!

Xqt added a comment.Feb 5 2020, 10:46 PM
In T242083#5854100, @Dvorapa wrote:

Okay, if DataSite.namespaces fails to get namespaces (due to timeout), this error is thrown, which is not a good behavior

This means a general exception is passed silently I guess.

Dvorapa added a comment.Feb 5 2020, 10:51 PM

It seems so as it then works with empty/incomplete namespace list and produces the error message in the description

Dvorapa mentioned this in T242081: Pywikibot fails to access Wikidata due to high maxlag lately.Feb 5 2020, 11:35 PM
Dvorapa added a subtask: T242081: Pywikibot fails to access Wikidata due to high maxlag lately.Feb 5 2020, 11:42 PM
Dvorapa added a comment.Mar 9 2020, 12:35 PM

@matej_suchanek Do you know any solution for this issue?

Summary: Wikidata maxlag produces empty Wikidata namespace list. Therefore self._entity_namespaces (which depends on it) is empty/incomplete and makes tests fail if that method to create self._entity_namespaces fails due to maxlag

matej_suchanek added a comment.Mar 9 2020, 12:47 PM
In T242083#5953475, @Dvorapa wrote:

@matej_suchanek Do you know any solution for this issue?

Failing hard in tests when maxlag is too high (no silent error and empty responses). Or not using (relaxing) maxlag on read requests.

Dvorapa added a comment.Mar 9 2020, 12:51 PM

@Xqt Could we somehow check whether any of self._entity_namespaces or WD namespaces list is (in)complete and try to recreate that if it isn't?

Perhaps we could check if WD namespaces list (cached) is empty or does not contain basic 0 to 15 namespaces?

Xqt added a comment.Mar 9 2020, 5:21 PM

Summary: Wikidata maxlag produces empty Wikidata namespace list

I am wondering. I get a TimeoutError in that case. And SkipTest should skip the test then too.
Where is the trick that the exception is not raisen? Probably there is no maxlag timeout from wikibase but just gives an empty or unsufficent NamespacesDict with _build_namespaces()

>>> import pwb, pywikibot as py
>>> s = py.Site()
>>> r = s.data_repository()
>>> ns = r.namespaces
Sleeping for 5.0 seconds, 2020-03-09 17:53:50
Sleeping for 5.5 seconds, 2020-03-09 17:53:55
Sleeping for 8.5 seconds, 2020-03-09 17:54:01
Sleeping for 11.3 seconds, 2020-03-09 17:54:10
Sleeping for 14.2 seconds, 2020-03-09 17:54:22
Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    ns = r.namespaces
  File "C:\pwb\GIT\core\pywikibot\site.py", line 1013, in namespaces
    self._namespaces = NamespacesDict(self._build_namespaces())
  File "C:\pwb\GIT\core\pywikibot\site.py", line 2647, in _build_namespaces
    for nsdata in self.siteinfo.get('namespaces', cache=False).values():
  File "C:\pwb\GIT\core\pywikibot\site.py", line 1683, in get
    preloaded = self._get_general(key, expiry)
  File "C:\pwb\GIT\core\pywikibot\site.py", line 1629, in _get_general
    default_info = self._get_siteinfo(props, expiry)
  File "C:\pwb\GIT\core\pywikibot\site.py", line 1552, in _get_siteinfo
    data = request.submit()
  File "C:\pwb\GIT\core\pywikibot\data\api.py", line 2258, in submit
    self._data = super(CachedRequest, self).submit()
  File "C:\pwb\GIT\core\pywikibot\data\api.py", line 2105, in submit
    raise MaxlagTimeoutError(msg)
pywikibot.exceptions.MaxlagTimeoutError: Maximum retries attempted due to maxlag without success.
Xqt added a comment.EditedMar 9 2020, 5:24 PM

if maxlag Timeout occurs the tests are skipped as expected (see wp.de and wp.en) but it seems there is not maxlag for ws.it. The NamespacesDict is insufficient instead:

Validate page_from_repository on wikipedia:de ... Sleeping for 5.0 seconds, 2020-03-09 16:11:17
Sleeping for 5.0 seconds, 2020-03-09 16:11:22
Sleeping for 5.0 seconds, 2020-03-09 16:11:28
Sleeping for 5.0 seconds, 2020-03-09 16:11:33
Sleeping for 6.2 seconds, 2020-03-09 16:11:38
skipped 'Maximum retries attempted due to maxlag without success.'
 26.987s test_page_from_repository_en_wp (tests.site_tests.TestCategoryFromWikibase)
Validate page_from_repository on wikipedia:en ... Sleeping for 5.0 seconds, 2020-03-09 16:11:44
Sleeping for 5.0 seconds, 2020-03-09 16:11:49
Sleeping for 5.0 seconds, 2020-03-09 16:11:54
Sleeping for 5.0 seconds, 2020-03-09 16:12:00
Sleeping for 7.1 seconds, 2020-03-09 16:12:05
skipped 'Maximum retries attempted due to maxlag without success.'
 27.878s test_page_from_repository_it_ws (tests.site_tests.TestCategoryFromWikibase)
Validate page_from_repository on wikisource:it ... ERROR

This is a new behaviour of api fault ihmo.

Xqt added a project: Upstream.Mar 10 2020, 7:41 AM
Xqt added a subscriber: hashar.Mar 22 2020, 1:17 PM
Xqt added a subscriber: Ladsgroup.Mar 25 2020, 2:53 PM
Dvorapa mentioned this in T245951: sparql_tests sometimes fails with TimeoutError.Apr 3 2020, 3:35 PM
Dvorapa moved this task from Backlog to CI test failures on the Pywikibot-tests board.Apr 3 2020, 4:05 PM
Xqt updated the task description. (Show Details)Apr 25 2020, 4:45 PM

@Ladsgroup: Any idea who can help upstream solving this malfunction?

Xqt added a project: MediaWiki-Action-API.Jun 11 2020, 5:15 PM
Restricted Application added a project: Platform Engineering. · View Herald TranscriptJun 11 2020, 5:15 PM
BPirkle removed a project: Platform Engineering.Jun 24 2020, 8:41 PM
hashar unsubscribed.Jun 25 2020, 7:26 AM
Dvorapa added a project: Platform Engineering.Jun 25 2020, 9:16 AM
In T242083#5954453, @Xqt wrote:

if maxlag Timeout occurs the tests are skipped as expected (see wp.de and wp.en) but it seems there is not maxlag for ws.it. The NamespacesDict is insufficient instead:

Validate page_from_repository on wikipedia:de ... Sleeping for 5.0 seconds, 2020-03-09 16:11:17
Sleeping for 5.0 seconds, 2020-03-09 16:11:22
Sleeping for 5.0 seconds, 2020-03-09 16:11:28
Sleeping for 5.0 seconds, 2020-03-09 16:11:33
Sleeping for 6.2 seconds, 2020-03-09 16:11:38
skipped 'Maximum retries attempted due to maxlag without success.'
 26.987s test_page_from_repository_en_wp (tests.site_tests.TestCategoryFromWikibase)
Validate page_from_repository on wikipedia:en ... Sleeping for 5.0 seconds, 2020-03-09 16:11:44
Sleeping for 5.0 seconds, 2020-03-09 16:11:49
Sleeping for 5.0 seconds, 2020-03-09 16:11:54
Sleeping for 5.0 seconds, 2020-03-09 16:12:00
Sleeping for 7.1 seconds, 2020-03-09 16:12:05
skipped 'Maximum retries attempted due to maxlag without success.'
 27.878s test_page_from_repository_it_ws (tests.site_tests.TestCategoryFromWikibase)
Validate page_from_repository on wikisource:it ... ERROR

This is a new behaviour of api fault ihmo.

Still waiting for a response

Dvorapa added a comment.Jun 25 2020, 9:18 AM

Asking ws:it API for namespaces gives broken list when maxlag condition occurs. No maxlag/timeout error, no empty list like in other cases.

• eprodromou edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.Jul 7 2020, 8:44 PM
• eprodromou removed a project: Platform Team Workboards (Clinic Duty Team).Jul 7 2020, 8:47 PM
• eprodromou subscribed.

What's the question that you're asking? Is there something we can do to help make this test pass?

Ladsgroup closed subtask T242081: Pywikibot fails to access Wikidata due to high maxlag lately as Resolved.Oct 21 2020, 5:17 PM
Xqt mentioned this in T266078: interwikidata.py throws pywikibot.data.api.APIError: param-missing: Either provide the Item "ids" or pairs of "sites" and "titles" for corresponding page.Oct 22 2020, 1:45 PM
Xqt closed this task as Resolved.Oct 23 2020, 9:17 AM
Xqt claimed this task.

Solved upstream with T242081 I guess

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL