Page MenuHomePhabricator
Create Task
Maniphest T258474

RDF dumps for Structured Data on Commons are broken
Closed, ResolvedPublic
Actions

Description

SDC should follow standard URI pattern for RDF dumps, but it does not.

The latest config change in this area was https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/609987/9/wmf-config/InitialiseSettings.php

Currently, wikidata's data and entity URI prefixes look like this:

@prefix wd: <http://www.wikidata.org/entity/> .
@prefix data: <https://www.wikidata.org/wiki/Special:EntityData/> .

Currently sdc has those prefixes.

@prefix sdc: <https://commons.wikimedia.org/wiki/Special:EntityData/> .
@prefix sdcdata: <https://commons.wikimedia.org/wiki/Special:EntityData/> .

With other prefixes build upon data prefix (or, as always, on entity prefix, but here they are identical).

sdc should be similar to wikidata:

@prefix sdc: <https://commons.wikimedia.org/entity/> .
@prefix sdcdata: <https://commons.wikimedia.org/wiki/Special:EntityData/> .

The correct prefixes should be:

@prefix sdc: <https://commons.wikimedia.org/entity/> .
@prefix sdcdata: <https://commons.wikimedia.org/wiki/Special:EntityData/> .
@prefix sdcs: <https://commons.wikimedia.org/entity/statement/> .
@prefix sdcref: <https://commons.wikimedia.org/reference/> .
@prefix sdcv: <https://commons.wikimedia.org/value/> .
@prefix sdct: <https://commons.wikimedia.org/prop/direct/> .
@prefix sdctn: <https://commons.wikimedia.org/prop/direct-normalized/> .
@prefix sdcp: <https://commons.wikimedia.org/prop/> .
@prefix sdcps: <https://commons.wikimedia.org/prop/statement/> .
@prefix sdcpsv: <https://commons.wikimedia.org/prop/statement/value/> .
@prefix sdcpsn: <https://commons.wikimedia.org/prop/statement/value-normalized/> .
@prefix sdcpq: <https://commons.wikimedia.org/prop/qualifier/> .
@prefix sdcpqv: <https://commons.wikimedia.org/prop/qualifier/value/> .
@prefix sdcpqn: <https://commons.wikimedia.org/prop/qualifier/value-normalized/> .
@prefix sdcpr: <https://commons.wikimedia.org/prop/reference/> .
@prefix sdcprv: <https://commons.wikimedia.org/prop/reference/value/> .
@prefix sdcprn: <https://commons.wikimedia.org/prop/reference/value-normalized/> .
@prefix sdcno: <https://commons.wikimedia.org/prop/novalue/> .

NT dumps are also affected.

Details

SubjectRepoBranchLines +/-
[sdoc] fix entity source base URIsoperations/mediawiki-configmaster+3 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Zbyszko created this task.Jul 21 2020, 8:22 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2020, 8:22 AM
dcausse renamed this task from RDF TTL format for Structured Data on Commons is broken to RDF dumps for Structured Data on Commons are broken.Jul 21 2020, 8:42 AM
dcausse triaged this task as High priority.
dcausse added projects: Wikidata-Query-Service, Wikidata.
dcausse updated the task description. (Show Details)
dcausse updated the task description. (Show Details)Jul 21 2020, 8:59 AM
dcausse added a comment.Jul 21 2020, 9:04 AM

I wonder if these IRIs should not be http instead of https because the Concept URI link on the file page refers to http://commons.wikimedia.org/entity/M123.

Addshore added a comment.Jul 21 2020, 9:21 AM
In T258474#6321939, @dcausse wrote:

I wonder if these IRIs should not be http instead of https because the Concept URI link on the file page refers to http://commons.wikimedia.org/entity/M123.

Having a decision there would be great :)
I don't know if the decision to go with https was a conscious one or not.

Addshore updated the task description. (Show Details)Jul 21 2020, 9:22 AM
Addshore added a project: Wikidata-Campsite.Jul 21 2020, 9:28 AM
Addshore added a project: [DEPRECATED] wdwb-tech.
Addshore added a project: User-Addshore.
dcausse added a comment.EditedJul 21 2020, 9:40 AM

I think the Concept URI link should match what's in the dump. T226453#5436469 suggests that if these URIs for commons are not yet too widespread https could be used.

dcausse updated the task description. (Show Details)Jul 21 2020, 10:03 AM
dcausse updated the task description. (Show Details)
dcausse updated the task description. (Show Details)Jul 21 2020, 10:26 AM
gerritbot added a comment.Jul 21 2020, 10:26 AM

Change 615171 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [sdoc] fix entity source base URIs

https://gerrit.wikimedia.org/r/615171

gerritbot added a project: Patch-For-Review.Jul 21 2020, 10:26 AM
dcausse updated the task description. (Show Details)Jul 21 2020, 12:32 PM
dcausse mentioned this in T258507: v: prefix not correctly prefixed in Wikibase when using entitysource config and extra prefixes.Jul 22 2020, 9:33 AM
dcausse added a subtask: T258507: v: prefix not correctly prefixed in Wikibase when using entitysource config and extra prefixes.Jul 22 2020, 9:39 AM
Addshore added a comment.Jul 22 2020, 11:55 AM

Patch looks good and IMO can be deployed whenever!

CBogen moved this task from Incoming to Current work on the Wikidata-Query-Service board.Jul 22 2020, 3:20 PM
CBogen added a project: Discovery-Search (Current work).
dcausse claimed this task.Jul 23 2020, 8:13 AM
dcausse moved this task from Incoming to To Be Deployed on the Discovery-Search (Current work) board.
gerritbot added a comment.Jul 23 2020, 11:05 AM

Change 615171 merged by jenkins-bot:
[operations/mediawiki-config@master] [sdoc] fix entity source base URIs

https://gerrit.wikimedia.org/r/615171

Maintenance_bot removed a project: Patch-For-Review.Jul 23 2020, 11:10 AM
Stashbot added a comment.Jul 23 2020, 11:13 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-23T11:13:23Z] <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T258474: [sdoc] fix entity source base URIs (duration: 01m 07s)

dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Jul 23 2020, 11:40 AM
Jheald subscribed.Aug 1 2020, 3:25 PM

Ticket description should be re-written.

SDC doesn't have its own properties, so prefixes like sdcp, sdcps etc are not appropriate and should not appear. (cf discussion at T258625)

dcausse added a comment.Aug 3 2020, 8:00 AM

@Jheald correct but in these tickets we always mentioned the list of all the sdc related prefixes seen in the dumps, the reason the dump is emitting all these "unneeded" prefixes is that it does not know at the time it writes the file header what are the list of effective prefixes it will need. I don't think it hurts to have unneeded prefixes declared in the turtle header.

Jheald added a comment.Aug 3 2020, 4:44 PM

@dcausse It *does* hurt a person who is trying to make sense of the dump, because they will see all these unfamiliar prefixes declared that they may then assume there will be corresponding kinds of predicates or objects that they have to make sense of.

Better to remove all the ones that we do not use, to make clearer the specific sdc ones that we do use.

Jheald mentioned this in T258625: Querying WCQS should allow me to use prefixes for MediaInfo items.Aug 3 2020, 4:48 PM
dcausse added a comment.Aug 4 2020, 7:45 AM

@Jheald I created T259587 for the problem you raised, the tickets here are for fixing the problems that make the dumps unusable. Unused prefixes, while confusing, are perfectly valid and won't cause a TTL parser to fail nor alter the meaning of the graph.

Gehel closed this task as Resolved.Aug 17 2020, 12:42 PM
Gehel closed subtask T258507: v: prefix not correctly prefixed in Wikibase when using entitysource config and extra prefixes as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits