Page MenuHomePhabricator
Create Task
Maniphest T311977

Wikimedia Commons entity dumps are lacking datatype field
Open, Needs TriagePublic
Actions

Description

I am working with Wikidata entity dumps and I am trying to use now also Wikimedia Commons entity dumps and I noticed that the latter are lacking the "datatype" field for statements. I see that there is JsonDataTypeInjector to inject that, but I assume it is not running for Wikimedia Commons entities dumps. Is there a reason why not? It would be great if both dumps would be as close to each other as possible (there will be still difference between statements and claims field, but that is easier to mitigate as it is a top-level field).

So, could we enable this injector? What does it take to do so?

Related Objects

Event Timeline

Mitar created this task.Jul 4 2022, 6:50 AM
Restricted Application added a project: Structured-Data-Backlog. · View Herald TranscriptJul 4 2022, 6:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Lucas_Werkmeister_WMDE added a comment.Jul 4 2022, 8:36 AM

I see that there is JsonDataTypeInjector to inject that, but I assume it is not running for Wikimedia Commons entities dumps.

I’m pretty sure it is running, but it only injects the datatype into claims/*, and as you mention, MediaInfo entities use statements/* instead (T149410). See also T246809: Inconsistencies between Wikidata and Structured Data about Snak's "datatype" from wbgetentities API results, which is the same issue as this but reported for wbgetentities (arguably you could merge these into one task, I think).

Mitar added a comment.Jul 4 2022, 10:56 AM

Oh, what a sad issue T149410. :-(

Anyway, this looks like it would be fixed by a simple fix of adding statements to globs here? Can I just go and submit a change adding that?

But I do not think fixing that would fix the wbgetentities API? Isn't JsonDataTypeInjector just for dumps?

CBogen moved this task from Triage to SDoC Statements on the Structured-Data-Backlog board.Jul 18 2022, 6:31 PM
LennardHofmann subscribed.Aug 3 2023, 7:29 PM
Mitar added a comment.Mar 7 2026, 8:45 PM

I cannot believe this is still a problem.

Can this be fixed by adding statements to globs, like I mentioned above? Can I submit a change?

Pppery added a project: Dumps-Generation.Mar 7 2026, 9:02 PM
Pppery subscribed.

Yes, you are welcome to submit a patch. See https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker

Restricted Application added a project: Data-Engineering. · View Herald TranscriptMar 7 2026, 9:02 PM
Pppery unsubscribed.Mar 7 2026, 9:02 PM
Mitar added a comment.Mar 7 2026, 9:10 PM

Thanks. My question is if we all agree that this is the right approach? Or do we wait for T149410?

Ottomata moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.Mar 11 2026, 4:44 PM
Ottomata added a project: Data-Engineering-Radar.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits