[epic] Add Structured data on Wikimedia Commons support to OpenRefine
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Spinster
	Aug 30 2021, 6:45 AM

Description

Add functionalities to OpenRefine that make it possible to

Edit and add StructuredDataOnCommons statements of/to Wikimedia Commons (MediaInfo) files
Upload new files with StructuredDataOnCommons to Wikimedia Commons (milestone, ETA end October 2022)

This is developed thanks to a Wikimedia grant.

More info: https://commons.wikimedia.org/wiki/Commons:OpenRefine

See also the equivalent issue and workboard on OpenRefine's GitHub, where OpenRefine-specific issues and development are tracked.

Timeline, planning and monthly brief reports of ongoing work here: https://meta.wikimedia.org/wiki/Grants:Project/CS&S/Structured_Data_on_Wikimedia_Commons_functionalities_in_OpenRefine/Timeline

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Spinster	T289971 [epic] Add Structured data on Wikimedia Commons support to OpenRefine
Stalled		None	T282796 Design a file format to represent Wikibase edits
Resolved		Spinster	T290296 Describe typical user workflows for batch editing and uploading structured data on Commons
Resolved		Loz.ross	T294938 Design research for Structured Data on Commons file upload in OpenRefine
Resolved		Loz.ross	T294939 Interview Wikimedia Commons / GLAM power users to understand their batch file upload workflows and needs
Resolved		Eugene233	T289803 Create a reconciliation service for (Structured Data on) Wikimedia Commons
Resolved		Spinster	T289808 Figure out and decide where to host the (Structured Data on) Wikimedia Commons reconciliation service
Open		None	T290088 The Structured Data on Commons reconciliation service recognizes the most widely used Commons file name formats
Invalid	Feature	None	T290089 Structured Data on Commons reconciliation service accepts Commons category names as input
Resolved		Eugene233	T290309 Set up repository for Commons reconciliation service
Resolved		Eugene233	T290310 Minimal prototype for the Commons recon service
Resolved		Eugene233	T293881 Support for all datatypes in the data extension service for Commons
Resolved		Eugene233	T299533 Support for monolingual text in the data extension service for Commons
Resolved		Eugene233	T299534 Support for Quantity datatype in the data extension service for Commons
Resolved		Eugene233	T293918 Property suggest service for the Commons reconciliation service
Resolved		Eugene233	T295802 Language of Commons Reconciliation Service should be visible in OpenRefine
Resolved		Spinster	T299134 Organize and hold first community meetup to demo and workshop SDC editing features in OpenRefine
Resolved		Alicia_Fagerving_WMSE	T301660 Blog post about OpenRefine development
Resolved		Spinster	T299135 Assemble various sets of interesting Commons files for testing SDC features in OpenRefine

Event Timeline

Spinster created this task.Aug 30 2021, 6:45 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 30 2021, 6:45 AM

Spinster triaged this task as Medium priority.Aug 30 2021, 6:46 AM

Spinster set Due Date to Jun 29 2022, 10:00 PM.

Spinster mentioned this in T289803: Create a reconciliation service for (Structured Data on) Wikimedia Commons.

Spinster added a subtask: T282796: Design a file format to represent Wikibase edits.Aug 30 2021, 6:52 AM

Spinster mentioned this in T282796: Design a file format to represent Wikibase edits.

Aklapper added a project: Epic.Aug 30 2021, 9:28 AM

Nikerabbit subscribed.Aug 30 2021, 1:20 PM

As a 'thing to keep in mind' - Commons will also support references in the (near) future per T230315: [XL] Create a way to see and add references to structured data on Commons (MediaInfo) statements

We can probably use https://commons.wikimedia.beta.wmflabs.org for testing, if that would be useful.
It seems to have most SDC-related functionalities (including Special:MediaSearch) but no full federation with Wikidata. For instance, try adding 'depicts octopus' to https://commons.wikimedia.beta.wmflabs.org/wiki/File:Otto_Puks.jpg.

Spinster awarded a token.Aug 31 2021, 6:46 AM

Spinster updated the task description. (Show Details)

Alicia_Fagerving_WMSE subscribed.Aug 31 2021, 9:48 AM

FRomeo_WMF added a subscriber: GFontenelle_WMF.Sep 2 2021, 6:03 PM

Spinster moved this task from SDC-support To Do (Minimum Viable Product) to SDC-support Doing on the OpenRefine board.Sep 3 2021, 6:51 AM

Spinster added a subtask: T290296: Describe typical user workflows for batch editing and uploading structured data on Commons.Sep 3 2021, 6:58 AM

Fuzheado subscribed.Sep 3 2021, 5:07 PM

MichellevL_WMNL subscribed.Sep 13 2021, 2:21 PM

Spinster updated the task description. (Show Details)Sep 14 2021, 7:21 AM

Spinster closed subtask T290296: Describe typical user workflows for batch editing and uploading structured data on Commons as Resolved.Nov 1 2021, 8:20 AM

Spinster added a subtask: T294939: Interview Wikimedia Commons / GLAM power users to understand their batch file upload workflows and needs.Nov 3 2021, 3:48 PM

Spinster removed a subtask: T294939: Interview Wikimedia Commons / GLAM power users to understand their batch file upload workflows and needs.

Spinster added a subtask: T294938: Design research for Structured Data on Commons file upload in OpenRefine.Nov 3 2021, 3:51 PM

Spinster added a subtask: T289803: Create a reconciliation service for (Structured Data on) Wikimedia Commons.

Spinster changed the status of subtask T294938: Design research for Structured Data on Commons file upload in OpenRefine from Open to In Progress.Nov 3 2021, 3:58 PM

Spinster added a subtask: T299134: Organize and hold first community meetup to demo and workshop SDC editing features in OpenRefine.Jan 13 2022, 1:48 PM

Spinster closed subtask T299135: Assemble various sets of interesting Commons files for testing SDC features in OpenRefine as Resolved.Jan 26 2022, 12:30 PM

Spinster closed subtask T299134: Organize and hold first community meetup to demo and workshop SDC editing features in OpenRefine as Resolved.Feb 24 2022, 1:48 PM

Loz.ross closed subtask T294938: Design research for Structured Data on Commons file upload in OpenRefine as Resolved.Apr 15 2022, 12:47 PM

Spinster closed subtask T289803: Create a reconciliation service for (Structured Data on) Wikimedia Commons as Resolved.Aug 11 2022, 12:50 PM

@Spinster: Hi, the Due Date set for this open task passed a while ago.
Could you please either update or reset the Due Date (by clicking Edit Task), or set the status of this task to resolved in case this task is done? Thanks!

No reply; resetting Due Date

Aklapper removed Due Date.Sep 12 2022, 7:10 AM

This task is nearly finished. The project got an extension until end October 2022, and around that time we hope to release an official OpenRefine version that supports file upload. (It is already possible via snapshot releases, see https://commons.wikimedia.org/wiki/Commons:OpenRefine)

Spinster updated the task description. (Show Details)Sep 12 2022, 3:10 PM

We're almost there! The upcoming OpenRefine 3.7 release will include full Wikimedia Commons upload functionalities.

We did this !!!! 🎉

OpenRefine 3.7 beta2 has been released with Wikimedia Commons editing and upload functionalities, and we also created a brand new Wikimedia Commons extension for OpenRefine that provides additional functionalities for Wikimedia Commons users (starting projects from Commons categories; specific GREL commands to process Wikitext; thumbnail previews).

Report for the Wikimedia project grant that made this possible: https://meta.wikimedia.org/wiki/Grants:Project/CS%26S/Structured_Data_on_Wikimedia_Commons_functionalities_in_OpenRefine/Final

Many thanks to everyone who worked very hard to make this a reality: @Pintoch @Eugene233 @Loz.ross and the many, many people who helped, advised, tested in some capacity, including but not limited to @Abbe98 @Alicia_Fagerving_WMSE @FRomeo_WMF @GFontenelle_WMF @OlafJanssen @Ainali