Page MenuHomePhabricator
Create Task
Maniphest T174870

isbn.py may fail in treat_page_and_item() with missing attribute claims for an ItemPage
Closed, ResolvedPublic
Actions

Description

=================================== FAILURES ===================================
________________________ TestIsbnWikibaseBot.test_isbn _________________________

self = <tests.isbn_tests.TestIsbnWikibaseBot testMethod=test_isbn>

    def test_isbn(self):
        """Test using the bot and wikibase."""
>       main('-page:' + self.test_page_qid, '-always', '-format')

tests/isbn_tests.py:243: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
scripts/isbn.py:1670: in main
    bot.run()
pywikibot/bot.py:1555: in run
    super(Bot, self).run()
pywikibot/bot.py:1464: in run
    self.treat(page)
pywikibot/bot.py:1755: in treat
    super(ExistingPageBot, self).treat(page)
pywikibot/bot.py:1678: in treat
    self.treat_page()
pywikibot/bot.py:2081: in treat_page
    self.treat_page_and_item(page, item)
scripts/isbn.py:1567: in treat_page_and_item
    if self.isbn_10_prop_id in item.claims:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ItemPage(Q943), name = 'claims'

    def __getattribute__(self, name):
        """Low-level attribute getter. Deprecates lastrevid."""
        if name == 'lastrevid':
            issue_deprecation_warning(
                'WikibasePage.lastrevid', 'latest_revision_id', 2)
            name = '_revid'
>       return super(WikibasePage, self).__getattribute__(name)
E       AttributeError: 'ItemPage' object has no attribute 'claims'

pywikibot/page.py:3677: AttributeError

Details

SubjectRepoBranchLines +/-
[bugfix] Check whether Claim has an "id" keypywikibot/coremaster+2 -2
[tests] split wikibase isbn tests and expect failure for the secondpywikibot/coremaster+7 -3
[bugfix] get ItemPage content in IsbnWikibaseBot.treat_page_and_item()pywikibot/coremaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Xqt created this task.Sep 2 2017, 10:22 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptSep 2 2017, 10:22 PM
Xqt triaged this task as High priority.Sep 2 2017, 10:22 PM
Dalba subscribed.Sep 3 2017, 12:08 AM

According to travis logs the test has started to fail after eb0ae50.

At the time the failiure has been:

TypeError: treat() missing 1 required positional argument: 'item'

But it changes to

AttributeError: 'ItemPage' object has no attribute 'claims'

after ea18c42fca51.

@matej_suchanek

Xqt added a comment.EditedSep 3 2017, 11:07 AM

maybe a missing ItemPage.get() ?
See https://gerrit.wikimedia.org/r/#/c/364435/4/pywikibot/bot.py@a2014

gerritbot subscribed.Sep 3 2017, 11:27 AM

Change 375612 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [bugfix] get ItemPage content in WikidataBot.treat_page()

https://gerrit.wikimedia.org/r/375612

gerritbot added a project: Patch-For-Review.Sep 3 2017, 11:27 AM
Xqt claimed this task.Sep 3 2017, 11:27 AM
Dalba added a comment.Sep 3 2017, 11:33 AM
In T174870#3575904, @gerritbot wrote:

Change 375612 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [bugfix] get ItemPage content in WikidataBot.treat_page()

https://gerrit.wikimedia.org/r/375612

Now I'm getting this error:

Traceback (most recent call last):
  File "...\Python36\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "...\Python36\lib\unittest\case.py", line 605, in run
    testMethod()
  File "...\pywikibot-core\tests\isbn_tests.py", line 245, in test_isbn
    main('-page:' + self.test_page_qid, '-always', '-to13')
  File "...\pywikibot-core\scripts\isbn.py", line 1670, in main
    bot.run()
  File "...\pywikibot-core\pywikibot\bot.py", line 1557, in run
    super(Bot, self).run()
  File "...\pywikibot-core\pywikibot\bot.py", line 1466, in run
    self.treat(page)
  File "...\pywikibot-core\pywikibot\bot.py", line 1757, in treat
    super(ExistingPageBot, self).treat(page)
  File "...\pywikibot-core\pywikibot\bot.py", line 1680, in treat
    self.treat_page()
  File "...\pywikibot-core\pywikibot\bot.py", line 2084, in treat_page
    self.treat_page_and_item(page, item)
  File "...\pywikibot-core\scripts\isbn.py", line 1628, in treat_page_and_item
    self.user_edit_entity(item, summary=self.comment)
  File "...\pywikibot-core\pywikibot\bot.py", line 1927, in user_edit_entity
    item._content if hasattr(item, '_content') else None))
  File "...\pywikibot-core\pywikibot\page.py", line 4289, in toJSON
    data = super(ItemPage, self).toJSON(diffto=diffto)
  File "...\pywikibot-core\pywikibot\page.py", line 3880, in toJSON
    claim_ids.add(claim['id'])
KeyError: 'id'
Xqt added a comment.Sep 3 2017, 4:03 PM

The old implementation had the KeyError too but the test didn't fail which is curious:

C:\pwb\GIT\core>py -2 pwb.py tests/isbn_tests -v TestIsbnWikibaseBot
tests: max_retries reduced from 15 to 1
test_isbn1 (__main__.TestIsbnWikibaseBot)
Test 1 using the bot and wikibase. ... Assuming that ISBN-10 property is P360.
Assuming that ISBN-13 property is P366.
Retrieving 1 pages from wikidata:test.


>>> Q943 <<<
Changing P360 (ISBN 097522980x --> 0-9752298-0-X)
>####
{u'id': u'Q943$fe3a849c-4c9f-2037-ae29-658443937c01',
 u'mainsnak': {u'datatype': u'string',
               u'datavalue': {u'type': u'string', u'value': u'0-9752298-0-X'},
               u'property': u'P360',
               u'snaktype': u'value'},
 u'rank': u'normal',
 u'type': u'statement'}
<####
{
    "claims": {
        "P360": [
            {
                "id": "Q943$fe3a849c-4c9f-2037-ae29-658443937c01",
                "mainsnak": {
                    "datatype": "string",
                    "datavalue": {
                        "type": "string",
                        "value": "0-9752298-0-X"
                    },
                    "property": "P360",
                    "snaktype": "value"
                },
                "rank": "normal",
                "type": "statement"
            }
        ]
    }
}
Change summary: Bot: Formatting ISBN
ok
test_isbn2 (__main__.TestIsbnWikibaseBot)
Test 1 using the bot and wikibase. ... WARNING: C:\pwb\GIT\core\pywikibot\bot.py
:856: UserWarning: Site objects have been created before arguments were handled
Assuming that ISBN-10 property is P360.
Assuming that ISBN-13 property is P366.
Retrieving 1 pages from wikidata:test.


>>> Q943 <<<
Changing P360 (ISBN 097522980x) to P366 (ISBN 978-0975229804)
>####
{u'mainsnak': {u'datatype': u'string',
               u'datavalue': {u'type': u'string',
                              u'value': u'ISBN 978-0975229804'},
               u'property': u'P366',
               u'snaktype': u'value'},
 u'rank': u'normal',
 u'type': u'statement'}
<####
ERROR: u'id'
Traceback (most recent call last):
  File "C:\pwb\GIT\core\pywikibot\bot.py", line 2030, in run
    self.treat(page, item)
  File "C:\pwb\GIT\core\scripts\isbn.py", line 1626, in treat
    self.user_edit_entity(item, summary=self.comment)
  File "C:\pwb\GIT\core\pywikibot\bot.py", line 1916, in user_edit_entity
    item._content if hasattr(item, '_content') else None))
  File "C:\pwb\GIT\core\pywikibot\page.py", line 4293, in toJSON
    data = super(ItemPage, self).toJSON(diffto=diffto)
  File "C:\pwb\GIT\core\pywikibot\page.py", line 3884, in toJSON
    claim_ids.add(claim['id'])
KeyError: u'id'ok

----------------------------------------------------------------------
Ran 2 tests in 5.347s

OK

C:\pwb\GIT\core>
Xqt added a comment.Sep 3 2017, 4:19 PM

I digged a bit deeper: The reason for not failing with the old implemetation was that WikidataBot.run() catched all exception and printed a warning only.

Dalba added a comment.Sep 3 2017, 4:24 PM
In T174870#3576025, @Xqt wrote:

The old implementation had the KeyError too but the test didn't fail which is curious:

I see. The run method of the old WikidataBot used to be inside a try-except clause:

    def run(self):
        """Process all pages in generator."""
...
        try:
....
        except Exception as e:
            pywikibot.exception(msg=e, tb=True)

That line (except Exception as e:) has been removed in 5bbd20c8d0311be74a1ea97b5705517be6b1170f .

Xqt added a comment.Sep 3 2017, 6:36 PM

That line (except Exception as e:) has been removed in 5bbd20c8d0311be74a1ea97b5705517be6b1170f .

Sure, but I guess it is wrong to pass exceptions silently. Test did also fail. I think there is a snak in the second test not a claim.

I propose to split two tests and let the second fail until the underlying problem is fixed.

Dalba added a comment.Sep 3 2017, 7:04 PM
In T174870#3576113, @Xqt wrote:

Sure, but I guess it is wrong to pass exceptions silently.

Agreed.

I think there is a snak in the second test not a claim.

I'm not familiar with wikibase terminology. But yes, I took a look and there was a claim for the ISBN-10 in Q943 which had an id, but there's also an ISBN-13 for the same ISBN-10 without any id. (I didn't check where it's coming from)

I propose to split two tests and let the second fail until the underlying problem is fixed.

Sounds OK to me.

gerritbot added a comment.Sep 3 2017, 7:07 PM

Change 375612 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] get ItemPage content in IsbnWikibaseBot.treat_page_and_item()

https://gerrit.wikimedia.org/r/375612

gerritbot added a comment.Sep 3 2017, 7:15 PM

Change 375623 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [tests] split wikibase isbn tests and expect failure for the second

https://gerrit.wikimedia.org/r/375623

Xqt added a parent task: T129368: Usages of unittest.expectedFailure or unittest.skip should be solved (goal).Sep 3 2017, 7:31 PM
gerritbot added a comment.Sep 3 2017, 7:43 PM

Change 375623 merged by jenkins-bot:
[pywikibot/core@master] [tests] split wikibase isbn tests and expect failure for the second

https://gerrit.wikimedia.org/r/375623

matej_suchanek edited projects, added TestMe; removed Patch-For-Review.Nov 5 2017, 2:27 PM
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptNov 5 2017, 2:27 PM
Dvorapa moved this task from Backlog to Test failures on the Pywikibot-tests board.May 20 2018, 11:59 AM
Xqt added a comment.EditedNov 3 2018, 2:35 PM

The behavior must have been changed. The new error is:

C:\pwb\GIT\core>pwb -lang:test isbn_tests -v TestIsbnWikibaseBot.test_isbn_to13
tests: max_retries reduced from 15 to 1
test_isbn_to13 (__main__.TestIsbnWikibaseBot)
Test to13 using the bot and wikibase. ... Q943
Assuming that ISBN-10 property is P360.
Assuming that ISBN-13 property is P366.
WARNING: IsbnWikibaseBot.__init__ set the Bot.site property; this is only needed when the Bot accesses many sites.
Retrieving 1 pages from wikidata:test.


>>> Q943 <<<
>>>>>>>>>>
Changing P360 (ISBN 097522980x) to P366 (ISBN 978-0975229804)

0 pages read
0 pages written
0 pages skipped
Execution time: 0 seconds
Script terminated by exception:

ERROR: KeyError: 'id'
ERROR

======================================================================
ERROR: test_isbn_to13 (__main__.TestIsbnWikibaseBot)
Test to13 using the bot and wikibase.
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".\tests\isbn_tests.py", line 291, in test_isbn_to13
    main('-page:' + self.test_page_qid, '-always', '-to13')
  File "C:\pwb\GIT\core\scripts\isbn.py", line 1677, in main
    bot.run()
  File "C:\pwb\GIT\core\pywikibot\bot.py", line 1625, in run
    super(Bot, self).run()
  File "C:\pwb\GIT\core\pywikibot\bot.py", line 1533, in run
    self.treat(page)
  File "C:\pwb\GIT\core\pywikibot\bot.py", line 1818, in treat
    self.treat_page()
  File "C:\pwb\GIT\core\pywikibot\bot.py", line 2274, in treat_page
    self.treat_page_and_item(page, item)
  File "C:\pwb\GIT\core\scripts\isbn.py", line 1635, in treat_page_and_item
    self.user_edit_entity(item, summary=self.comment)
  File "C:\pwb\GIT\core\pywikibot\bot.py", line 2057, in user_edit_entity
    diff = item.toJSON(diffto=(
  File "C:\pwb\GIT\core\pywikibot\page.py", line 4660, in toJSON
    data = super(ItemPage, self).toJSON(diffto=diffto)
  File "C:\pwb\GIT\core\pywikibot\page.py", line 4157, in toJSON
    claim_ids.add(claim['id'])
KeyError: 'id'

----------------------------------------------------------------------
Ran 1 test in 2.241s

FAILED (errors=1)

C:\pwb\GIT\core>
xSavitar moved this task from Backlog to Needs Review on the Pywikibot board.Nov 5 2018, 11:28 AM
Xqt raised the priority of this task from High to Needs Triage.Feb 3 2019, 12:28 PM
Xqt moved this task from Needs Review to Backlog on the Pywikibot board.
Xqt triaged this task as High priority.Feb 4 2019, 11:33 AM
gerritbot added a comment.Jan 19 2020, 3:09 PM

Change 565768 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [bugfix] Check whether Claim has an "id" key

https://gerrit.wikimedia.org/r/565768

gerritbot added a project: Patch-For-Review.Jan 19 2020, 3:09 PM
gerritbot added a comment.Jan 20 2020, 3:04 PM

Change 565768 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] Check whether Claim has an "id" key

https://gerrit.wikimedia.org/r/565768

Xqt mentioned this in rPWBC6cdbc2565d67: [bugfix] Check whether Claim has an "id" key.Jan 20 2020, 3:05 PM
Xqt closed this task as Resolved.Jan 20 2020, 3:33 PM
matej_suchanek removed projects: Patch-For-Review, TestMe.Feb 3 2020, 3:54 PM
matej_suchanek unsubscribed.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits