Page MenuHomePhabricator

qualifier misidentified in fromJSON()
Closed, ResolvedPublic

Description

Currently qualifiers are misidentified as references when fromJSON() is called on the parent claim.

To test:

$ python pwb.py shell
Welcome to the Pywikibot interactive shell!
>>> repo = pywikibot.Site('wikidata', 'wikidata').data_repository()
>>> item = pywikibot.ItemPage(repo, 'Q4115189')
>>> item.exists()
True
>>> c = item.claims['P800'][0]
>>> sc = c.sources[1]['P813'][0]
>>> qc = c.qualifiers['P580'][0]
>>> qc.isReference
True
>>> qc.isQualifier
False
>>> sc.isReference
True
>>> sc.isQualifier
False

https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q4115189 contains

...
                "P800": [
                    {
                        "mainsnak": {
                            "snaktype": "value",
                            "property": "P800",
                            "datavalue": {
                                "value": {
                                    "entity-type": "item",
                                    "numeric-id": 4115189
                                },
                                "type": "wikibase-entityid"
                            },
                            "datatype": "wikibase-item"
                        },
                        "type": "statement",
                        "qualifiers": {
                            "P580": [
                                {
                                    "snaktype": "value",
                                    "property": "P580",
                                    "hash": "a1e9eb581e4e1ce68f837f4e7f2751373180abd8",
                                    "datavalue": {
                                        "value": {
                                            "time": "+1901-02-03T00:00:00Z",
                                            "timezone": 0,
                                            "before": 0,
                                            "after": 0,
                                            "precision": 11,
                                            "calendarmodel": "http://www.wikidata.org/entity/Q1985727"
                                        },
                                        "type": "time"
                                    },
                                    "datatype": "time"
                                }
                            ]
                        },
                        "qualifiers-order": [
                            "P580"
                        ],
                        "id": "Q4115189$e3d28f24-4982-a547-985d-136e972fd58a",
                        "rank": "normal",
                        "references": [
                            {
                                "hash": "7ad766a38be5f2a6af31d4dffea0ba491be0e0bf",
                                "snaks": {
                                    "P50": [
                                        {
                                            "snaktype": "value",
                                            "property": "P50",
                                            "datavalue": {
                                                "value": {
                                                    "entity-type": "item",
                                                    "numeric-id": 14411
                                                },
                                                "type": "wikibase-entityid"
                                            },
                                            "datatype": "wikibase-item"
                                        }
                                    ]
                                },
                                "snaks-order": [
                                    "P50"
                                ]
                            },
                            {
                                "hash": "f172f7e0b94254fcdaaf9027714322a06e9872a8",
                                "snaks": {
                                    "P813": [
                                        {
                                            "snaktype": "value",
                                            "property": "P813",
                                            "datavalue": {
                                                "value": {
                                                    "time": "+2014-11-29T00:00:00Z",
                                                    "timezone": 0,
                                                    "before": 0,
                                                    "after": 0,
                                                    "precision": 11,
                                                    "calendarmodel": "http://www.wikidata.org/entity/Q1985727"
                                                },
                                                "type": "time"
                                            },
                                            "datatype": "time"
                                        }
                                    ],
                                    "P854": [
                                        {
                                            "snaktype": "value",
                                            "property": "P854",
                                            "datavalue": {
                                                "value": "https://www.wikidata.org/wiki/Q4115189",
                                                "type": "string"
                                            },
                                            "datatype": "url"
                                        }
                                    ]
                                },
                                "snaks-order": [
                                    "P813",
                                    "P854"
                                ]
                            }
                        ]
                    }
                ],
...

There we can see hash in both qualifiers and references, however the hash is at a different 'level' in the JSON.

I have a quick patch which simply moves the isQualifier/isReference assignment to the qualifierFromJSON()/referenceFromJSON() methods. However since I'm unsure if fromJSON() is ever called directly on a qualifer/reference I'm not sure whether this is enough.

Event Timeline

Lokal_Profil raised the priority of this task from to Needs Triage.
Lokal_Profil updated the task description. (Show Details)
Lokal_Profil added a project: Pywikibot.
Lokal_Profil subscribed.

Change 239780 had a related patch set uploaded (by Lokal Profil):
Correctly identify qualifier from JSON

https://gerrit.wikimedia.org/r/239780

To clarify. The source of the above error is an false assumption that qualifiers don't have a hash property.

jayvdb set Security to None.

Interesting; toJSON contains qualifier.isQualifier = True and source.isReference = True. Those should be asserts!

The bug with regards to qualifiers vs references objects slipped in during cc62fbbac21f0 , which changed qualifierFromJSON so that fromJSON was called with hash provided.

Prior to that, since ea33047d, fromJSON always set isQualifier = True even for primary claims which were neither qualifier or reference, but that is a different bug.

I agree with the patch, which solves both bugs. Anyone calling fromJSON for a reference or a qualifier should have been using referenceFromJSON or qualifierFromJSON, which have existed for over two years, so there is no excuse for using fromJSON with a reference or a qualifier.

Change 239780 merged by jenkins-bot:
Correctly identify qualifier from JSON

https://gerrit.wikimedia.org/r/239780

Interesting; toJSON contains qualifier.isQualifier = True and source.isReference = True. Those should be asserts!

Should we patch this as well before resolving this task?

Also how is pywikibot-wikibase handled? Should pulls be done there as well or will that synk up before it goes live (assuming that it is not already)

Change 243043 had a related patch set uploaded (by John Vandenberg):
Correctly identify qualifier from JSON

https://gerrit.wikimedia.org/r/243043

Interesting; toJSON contains qualifier.isQualifier = True and source.isReference = True. Those should be asserts!

Should we patch this as well before resolving this task?

Yes please.

Also how is pywikibot-wikibase handled? Should pulls be done there as well or will that synk up before it goes live (assuming that it is not already)

I have no idea about pywikibot-wikibase. It is @Ladsgroup's fork of some pywikibot code. A quick peak indicates it also has this bug.

Change 243043 merged by jenkins-bot:
Correctly identify qualifier from JSON

https://gerrit.wikimedia.org/r/243043

Change 244834 had a related patch set uploaded (by John Vandenberg):
toJSON assert correct claim type

https://gerrit.wikimedia.org/r/244834

Make a patch in pywikibase (pywikibot-wikibase) and I will merge it. To give credit to you.
My plan is to move non-editing parts of pywikibot related to wikibase into another repo and use it from there. Your help would be really appreciated.

Change 256210 had a related patch set uploaded (by Lokal Profil):
Correctly identify qualifier from JSON

https://gerrit.wikimedia.org/r/256210

Change 256210 had a related patch set uploaded (by Lokal Profil):
Correctly identify qualifier from JSON

https://gerrit.wikimedia.org/r/256210

The above is the patch for pywikibot-wikibase

If someone can resolve https://gerrit.wikimedia.org/r/244834 (deciding between if assert qualifier.isQualifier and assert qualifier.isQualifier is True) then I think we could close this task.

Change 244834 merged by jenkins-bot:
toJSON assert correct claim type

https://gerrit.wikimedia.org/r/244834

jayvdb assigned this task to Lokal_Profil.

Change 256210 merged by jenkins-bot:
Correctly identify qualifier from JSON

https://gerrit.wikimedia.org/r/256210