Page MenuHomePhabricator
Create Task
Maniphest T277665

[L] Determine an IRI to join commons mediainfo entities and wikidata properties referencing commons images
Closed, ResolvedPublic
Actions

Description

As a user of WDQS and WCQS I want to be able to join a mediainfo item with a property value referencing a commons image such as P18.

There are no obvious ways to do this currently.

In the RDF output wikidata the properties instance of Q18610173 (e.g. P18) do use an IRI in the form:
http://commons.wikimedia.org/wiki/Special:FilePath/_filename_ while commons entities reference their contentUrl using https://upload.wikimedia.org/wikipedia/commons/X/XY/_filename.

These IRIs could have been used for joining but they are different:

  • the use of Special:FilePath
  • the use of http vs https

There should exist a common IRI identifying a commons file.

With a commons RDF output like:

sdc:M10031710 a wikibase:Mediainfo,
		schema:MediaObject,
		schema:ImageObject ;
	schema:encodingFormat "image/jpeg" ;
	schema:contentUrl <https://upload.wikimedia.org/wikipedia/commons/c/c0/Douglas_adams_portrait_cropped.jpg> ;

One approach could be to change the URL emitted by commons mediainfo to be the same as the one by wikidata. This is a breaking approach: 

sdc:M10031710 a wikibase:Mediainfo,
		schema:MediaObject,
		schema:ImageObject ;
	schema:encodingFormat "image/jpeg" ;
	schema:contentUrl <http://commons.wikimedia.org/wiki/Special:FilePath/Douglas%20adams%20portrait%20cropped.jpg> ;

Another (preferred) approach would to introduce a new triple, e.g. schema:url. This would increase the size of the graph, but by an acceptable amount, and without adding any breaking changes:

sdc:M10031710 a wikibase:Mediainfo,
		schema:MediaObject,
		schema:ImageObject ;
	schema:encodingFormat "image/jpeg" ;
	schema:contentUrl <https://upload.wikimedia.org/wikipedia/commons/c/c0/Douglas_adams_portrait_cropped.jpg> ;
	schema:url <http://commons.wikimedia.org/wiki/Special:FilePath/Douglas%20adams%20portrait%20cropped.jpg> ;

Note on similar tickets:

  • T258769 is very similar but is worded to simplify the use of the image grid feature of the UI
  • T258776 serves similar purposes but is harder to achieve:
    • it requires "synchronization" between wikidata and commons to obtain the page ID of a media info item
    • MediaInfo item may not yet exist while the commons image is still referenceable from wikidata

AC:

  • A query on WCQS for joining wikidata item from WDQS using federation can be easily written without complex string manipulation
  • A new schema:url triple is added

Details

SubjectRepoBranchLines +/-
Determine an IRI to join commons mediainfo entities and wikidata propertiesmediawiki/extensions/WikibaseMediaInfomaster+14 -0
Customize query in gerrit

Related Objects

Event Timeline

• dcausse created this task.Mar 17 2021, 2:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 17 2021, 2:55 PM
• dcausse mentioned this in T258769: ImageGrid for WCQS.Mar 17 2021, 2:58 PM
• dcausse mentioned this in T258776: Add Structured Data on Commons M-ID to Wikidata dumps.
• dcausse updated the task description. (Show Details)Mar 17 2021, 3:02 PM
CBogen updated the task description. (Show Details)Mar 17 2021, 3:18 PM
CBogen merged a task: T258769: ImageGrid for WCQS.
CBogen added subscribers: Gehel, Ainali, Nikki and 6 others.
CBogen merged a task: T258776: Add Structured Data on Commons M-ID to Wikidata dumps.Mar 17 2021, 3:20 PM
CBogen added subscribers: Tpt, CBogen, Fuzheado and 7 others.
• dcausse updated the task description. (Show Details)Mar 19 2021, 3:32 PM
Multichill added a comment.Mar 20 2021, 11:44 AM

File names are bad URI's. Files get renamed all the time (see https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=&limit=500&type=move ) causing all sorts of breakage. The pageid stays the same so the mediaid also stays the same. That's a much more stable identifier.

Rachmat04 unsubscribed.Mar 20 2021, 11:57 AM
CBogen edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.Mar 30 2021, 3:09 PM
CBogen moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.
CBogen renamed this task from Determine an IRI to join commons mediainfo entities and wikidata properties referencing commons images to [L] Determine an IRI to join commons mediainfo entities and wikidata properties referencing commons images.Apr 7 2021, 4:43 PM
CBogen moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.Apr 7 2021, 4:59 PM
Lokal_Profil subscribed.Apr 8 2021, 10:54 PM
Seddon claimed this task.Jun 22 2021, 7:34 PM
Seddon moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.
gerritbot added a comment.Jul 15 2021, 11:55 PM

Change 704881 had a related patch set uploaded (by Seddon; author: Seddon):

[mediawiki/extensions/WikibaseMediaInfo@master] WIP Determine an IRI to join commons mediainfo entities and wikidata properties

https://gerrit.wikimedia.org/r/704881

gerritbot added a project: Patch-For-Review.Jul 15 2021, 11:55 PM
Seddon added a comment.Jul 19 2021, 8:45 AM

Just to note that this ticket will require a config patch prior to deployment.

CBogen moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Jul 26 2021, 4:21 PM
Cparle moved this task from Code Review to Verify on Production on the Structured-Data-Backlog (Current Work) board.Jul 30 2021, 3:00 PM
gerritbot added a comment.Jul 30 2021, 3:18 PM

Change 704881 merged by jenkins-bot:

[mediawiki/extensions/WikibaseMediaInfo@master] Determine an IRI to join commons mediainfo entities and wikidata properties

https://gerrit.wikimedia.org/r/704881

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.17; 2021-08-02).Jul 30 2021, 4:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 30 2021, 4:10 PM
Multichill added a comment.Jul 31 2021, 10:18 AM

I expect encoding issues. How are you handling spaces and underscores for example? https://commons.wikimedia.org/wiki/File:Stilleven,_SK-C-1561.jpg gives <schema:contentUrl rdf:resource="https://upload.wikimedia.org/wikipedia/commons/f/f4/Stilleven%2C_SK-C-1561.jpg"/> in https://commons.wikimedia.org/wiki/Special:EntityData/M83614923.rdf and <ps:P18 rdf:resource="http://commons.wikimedia.org/wiki/Special:FilePath/Stilleven%2C%20SK-C-1561.jpg"/> in https://www.wikidata.org/wiki/Special:EntityData/Q18608269.rdf .

Would probably be good to add some unit tests for these kind of cases.

So9q subscribed.Aug 9 2021, 1:18 PM
So9q added a comment.EditedAug 9 2021, 1:20 PM
In T277665#6931281, @Multichill wrote:

File names are bad URI's. Files get renamed all the time (see https://commons.wikimedia.org/w/index.php?title=Special:Log&offset=&limit=500&type=move ) causing all sorts of breakage. The pageid stays the same so the mediaid also stays the same. That's a much more stable identifier.

+1

So9q mentioned this in T288462: Filenames are unstable on Commons. Switch to MID datatype on P18 and other properties?.Aug 9 2021, 1:25 PM
Seddon moved this task from Verify on Production to Doing on the Structured-Data-Backlog (Current Work) board.Aug 9 2021, 3:53 PM
Seddon added a comment.Aug 11 2021, 12:21 PM
In T277665#7250281, @Multichill wrote:

I expect encoding issues. How are you handling spaces and underscores for example? https://commons.wikimedia.org/wiki/File:Stilleven,_SK-C-1561.jpg gives <schema:contentUrl rdf:resource="https://upload.wikimedia.org/wikipedia/commons/f/f4/Stilleven%2C_SK-C-1561.jpg"/> in https://commons.wikimedia.org/wiki/Special:EntityData/M83614923.rdf and <ps:P18 rdf:resource="http://commons.wikimedia.org/wiki/Special:FilePath/Stilleven%2C%20SK-C-1561.jpg"/> in https://www.wikidata.org/wiki/Special:EntityData/Q18608269.rdf .

Would probably be good to add some unit tests for these kind of cases.

Joining between Wikimedia Commons and Wikidata via P18 in the WCQS can be done using schema:url going forward. Both are using the same encoding and should match.

<schema:url rdf:resource="http://commons.wikimedia.org/wiki/Special:FilePath/Stilleven%2C%20SK-C-1561.jpg"/>
<ps:P18 rdf:resource="http://commons.wikimedia.org/wiki/Special:FilePath/Stilleven%2C%20SK-C-1561.jpg"/>

Seddon updated the task description. (Show Details)Aug 11 2021, 12:23 PM
Seddon added a comment.EditedAug 12 2021, 1:04 PM

Based on this query it seems to working for WCQS/WMQS queries.

Seddon closed this task as Resolved.Aug 12 2021, 1:05 PM
Seddon updated the task description. (Show Details)
Husky added a comment.Aug 13 2021, 9:18 AM

Yeah, great! This simplifies the queries so much, without that awful hack of using the string concat method. Finding all images depicting Douglas Adams is now just:

#defaultView:ImageGrid
select ?file ?image where {
  ?file wdt:P180 wd:Q42;
        schema:url ?image.
}
• dcausse mentioned this in T288230: Promote MediaInfo RDF format to stable.Aug 26 2021, 8:10 AM
Multichill added a comment.Sep 17 2021, 3:06 PM
In T277665#7275279, @Seddon wrote:

Joining between Wikimedia Commons and Wikidata via P18 in the WCQS can be done using schema:url going forward. Both are using the same encoding and should match.

<schema:url rdf:resource="http://commons.wikimedia.org/wiki/Special:FilePath/Stilleven%2C%20SK-C-1561.jpg"/>
<ps:P18 rdf:resource="http://commons.wikimedia.org/wiki/Special:FilePath/Stilleven%2C%20SK-C-1561.jpg"/>

That should be httpS on both sides.

• dcausse mentioned this in T344876: Wikibase MediaInfo should provide access to page name via query service.Sep 7 2023, 7:42 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits