Page MenuHomePhabricator
Create Task
Maniphest T289803

Create a reconciliation service for (Structured Data on) Wikimedia Commons
Closed, ResolvedPublic
Actions

Description

Develop a reconciliation service for structured data on Wikimedia Commons (SDC / StructuredDataOnCommons) which, just like the Wikidata reconciliation service, conforms to the Reconciliation Service API / protocol. This reconciliation service will be used by OpenRefine and can also be used by other tools.

This will be developed as part of the Structured Data on Commons Functionalities in OpenRefine project. Development work is planned ~ September-November 2021. Also see T289971: [epic] Add Structured data on Wikimedia Commons support to OpenRefine.

Functionalities to include:

  • Take a list of file names from Wikimedia Commons and convert these file names to their corresponding entity identifiers (“M numbers” or M-ids - the Wikimedia Commons equivalent of Q-ids)
  • Provide a data extension service that fetches
    • values of requested properties of the media files
    • wikitext of the media files (preferably parsed in a semi-structured way - interestingly, this is NOT structured data, but is necessary to allow further data operations in OpenRefine)
    • categories of the media files (ditto) (via Wikitext)
    • the existing structured data statements (if any exist) of the MediaInfo entity
    • the existing captions of the MediaInfo entity

Related Objects
Search...

Event Timeline

Spinster created this task.Aug 26 2021, 6:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2021, 6:05 PM
Spinster triaged this task as Medium priority.Aug 26 2021, 6:09 PM
Spinster set Due Date to Oct 28 2021, 10:00 PM.
Spinster updated the task description. (Show Details)
Spinster updated the task description. (Show Details)
Spinster moved this task from Backlog to SDC-support To Do (Minimum Viable Product) on the OpenRefine board.Aug 26 2021, 7:20 PM
Spinster updated the task description. (Show Details)Aug 26 2021, 7:29 PM
Spinster updated the task description. (Show Details)Aug 30 2021, 6:48 AM

This is developed as part of T289971: [epic] Add Structured data on Wikimedia Commons support to OpenRefine but will be useful to more tools/software than just OpenRefine.

Pintoch subscribed.Aug 30 2021, 9:39 AM

If we want to host this project on GitHub, it could be done here: https://github.com/OpenRefine/commons-recon-service
(But we could also use Wikimedia's Gerrit).

Aklapper removed a subscriber: Reconciliation.Sep 1 2021, 12:56 PM
Spinster added a comment.Sep 1 2021, 2:55 PM
In T289803#7318025, @Pintoch wrote:

If we want to host this project on GitHub, it could be done here: https://github.com/OpenRefine/commons-recon-service
(But we could also use Wikimedia's Gerrit).

In today's team kickoff meeting, we decided we'll host it on Gerrit.

Spinster assigned this task to Eugene233.Sep 1 2021, 2:55 PM
FRomeo_WMF added a subscriber: GFontenelle_WMF.Sep 2 2021, 6:04 PM
FRomeo_WMF subscribed.
Spinster moved this task from SDC-support To Do (Minimum Viable Product) to SDC-support Doing on the OpenRefine board.Sep 3 2021, 6:51 AM
Pintoch added a subtask: T290309: Set up repository for Commons reconciliation service.Sep 3 2021, 9:34 AM
MichellevL_WMNL subscribed.Sep 13 2021, 2:23 PM
Eugene233 closed subtask T290309: Set up repository for Commons reconciliation service as Resolved.Sep 17 2021, 7:21 AM
Spinster closed subtask T289808: Figure out and decide where to host the (Structured Data on) Wikimedia Commons reconciliation service as Resolved.Oct 6 2021, 6:45 AM
Pintoch closed subtask T290310: Minimal prototype for the Commons recon service as Resolved.Oct 20 2021, 11:31 AM
Spinster added a parent task: T289971: [epic] Add Structured data on Wikimedia Commons support to OpenRefine.Nov 3 2021, 3:52 PM
Spinster added a subtask: T295802: Language of Commons Reconciliation Service should be visible in OpenRefine.Nov 16 2021, 7:13 PM
Pintoch closed subtask T293918: Property suggest service for the Commons reconciliation service as Resolved.Dec 2 2021, 2:02 PM
Aklapper added a comment.Dec 6 2021, 9:31 AM

@Spinster / @Eugene233: Hi, the Due Date set for this open task passed a while ago.
Could you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!

Spinster removed Due Date.Dec 6 2021, 12:50 PM
In T289803#7548965, @Aklapper wrote:

@Spinster / @Eugene233: Hi, the Due Date set for this open task passed a while ago.
Could you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!

I removed the due date :-) We are quite far on this task, still ironing out some bugs.

Spinster closed subtask T293881: Support for all datatypes in the data extension service for Commons as Resolved.Feb 23 2022, 4:00 PM
Spinster closed subtask T295802: Language of Commons Reconciliation Service should be visible in OpenRefine as Resolved.Feb 23 2022, 4:06 PM
Pintoch added a comment.Jul 9 2022, 8:58 PM

Should we close this as done?

Eugene233 added a comment.Jul 12 2022, 11:16 AM

@Pintoch We still need some cleaning up to be done IMO. Should be creating a few tasks around this.

Pintoch added a comment.Jul 12 2022, 11:20 AM

Sounds good! I think @Spinster also intends to write a few follow-up tickets.

Spinster closed this task as Resolved.Aug 11 2022, 12:49 PM
Spinster updated the task description. (Show Details)

This main task is totally done, so I will close it! Should have done this months ago already. Like with every other tool, there will always be lingering and new things to fix, but the basics are totally in place 💃

GFontenelle_WMF awarded a token.Aug 11 2022, 1:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL