Page MenuHomePhabricator
Create Task
Maniphest T220525

MCR: Import all slots from XML dumps
Closed, ResolvedPublic
Actions

Description

Once we have T174031: MCR: Include all slots in XML dumps, we need to also be able to read/import slots other than the main slot from dumps. This means implementing support for XML schema version 0.11 in WikiImporter. To enable this, we'll probably want to turn WikiRevision into a wrapper for MutableRevisionRecord.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

daniel created this task.Apr 9 2019, 3:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 9 2019, 3:57 PM
daniel added a comment.Apr 9 2019, 3:58 PM

I just realized we never did that. It seems kind of important ;)

CCicalese_WMF triaged this task as Medium priority.Apr 17 2019, 7:49 PM
CCicalese_WMF edited projects, added Platform Engineering (MCR), Platform Team Legacy (Next); removed Platform Engineering.
CCicalese_WMF removed a project: Platform Team Legacy (Next).Jul 14 2019, 3:16 PM
• WDoranWMF moved this task from MCR to mop on the Platform Engineering board.Jul 26 2019, 6:38 PM
• WDoranWMF edited projects, added Core Platform Team Initiatives (MCR); removed Platform Engineering (MCR).
• Pchelolo claimed this task.Aug 23 2019, 6:45 PM
• Nuria subscribed.Nov 21 2019, 10:24 PM

From other tickets i gather that there was an agreement for a format that would contain all slots, seems like this one was version 0.11 https://www.mediawiki.org/wiki/Requests_for_comment/Schema_update_for_multiple_content_objects_per_revision_(MCR)_in_XML_dumps#Schema

But the version of the current dumps on commons are version="0.10" so, are there any dumps that include the slots that in turn include the structure data?

• Nuria added a comment.Nov 21 2019, 10:24 PM

cc @ArielGlenn which might know the answer to the question

daniel added a subscriber: CCicalese_WMF.Nov 21 2019, 10:46 PM

@Nuria 0.10 is still the default format, we should probably change that. Maybe this could 3even make it into 1.34 still, I suppose we just forgot to move it forward. @CCicalese_WMF, thoughts?

• Nuria added a comment.Nov 21 2019, 10:48 PM

@daniel: so I understand since i know little about all this. At this time the slots that contain the structure data items on say, a page in commons, are NOT included in the dumps with the page itself. Correct?

Is that structure data being dumped elsewhere on its own?

• Nuria mentioned this in T238878: Data about how many file pages on Commons contain at least one structured data element .Nov 21 2019, 11:27 PM
ArielGlenn added a comment.Nov 22 2019, 7:09 AM
In T220525#5683009, @Nuria wrote:

@daniel: so I understand since i know little about all this. At this time the slots that contain the structure data items on say, a page in commons, are NOT included in the dumps with the page itself. Correct?

Is that structure data being dumped elsewhere on its own?

Not yet; there's a task for that but it's blocked on a performance issue. See https://phabricator.wikimedia.org/T222497 the blocker, and https://phabricator.wikimedia.org/T221917 the dumps task.

daniel added subscribers: • Ramsey-WMF, Abit.Nov 22 2019, 11:38 AM
In T220525#5683009, @Nuria wrote:

@daniel: so I understand since i know little about all this. At this time the slots that contain the structure data items on say, a page in commons, are NOT included in the dumps with the page itself. Correct?

Is that structure data being dumped elsewhere on its own?

Data in slots other than the main slot are not dumped anywhere right now. This was tagged as Not A Blocker (tm) for the MVP. Ask @Abit and @Ramsey-WMF about the reasoning.

daniel added a comment.EditedNov 22 2019, 11:39 AM
In T220525#5683597, @ArielGlenn wrote:

Not yet; there's a task for that but it's blocked on a performance issue. See https://phabricator.wikimedia.org/T222497 the blocker, and https://phabricator.wikimedia.org/T221917 the dumps task.

To clarify - the blocker is for the RDF dumps. Including the MediaInfo slot in the XML dump is not blocked on anything, we could just do it. Or am I missing something?

ArielGlenn added a comment.Nov 22 2019, 11:43 AM
In T220525#5684216, @daniel wrote:
In T220525#5683597, @ArielGlenn wrote:

Not yet; there's a task for that but it's blocked on a performance issue. See https://phabricator.wikimedia.org/T222497 the blocker, and https://phabricator.wikimedia.org/T221917 the dumps task.

To clarify - the blocker is for the RDF dumps. Including the MediaInfo slot in the XML dump is not blocked on anything, we could just do it. Or am I missing something?

That's right, this is an answer to the question "Is that structured data being dumped elsewhere on its own" (like the wikidata entity dumps).

• Nuria added a project: Analytics.Nov 22 2019, 5:57 PM
ArielGlenn added a project: User-ArielGlenn.Nov 22 2019, 6:24 PM
JAllemandou subscribed.Nov 25 2019, 5:14 PM
mforns moved this task from Incoming to Radar on the Analytics board.Nov 25 2019, 5:14 PM
CCicalese_WMF edited projects, added Core Platform Team Initiatives (MCR Schema Migration); removed Core Platform Team Initiatives (MCR).Dec 19 2019, 6:36 PM
daniel added a project: Platform Team Workboards (Clinic Duty Team).Apr 15 2020, 4:03 PM

Putting this con the CPT clinic duty board as a "small project".

• Pchelolo removed • Pchelolo as the assignee of this task.Apr 15 2020, 4:04 PM
• Pchelolo subscribed.
• Pchelolo assigned this task to CCicalese_WMF.Apr 15 2020, 4:47 PM
• Pchelolo moved this task from Inbox to Ready (WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.
Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:33 AM
CCicalese_WMF added a parent task: T174031: MCR: Include all slots in XML dumps.Jul 2 2020, 10:36 PM
gerritbot added a comment.Jul 13 2020, 8:54 PM

Change 586316 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] WIP: MCR import

https://gerrit.wikimedia.org/r/586316

gerritbot added a project: Patch-For-Review.Jul 13 2020, 8:54 PM

Change 612417 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[mediawiki/extensions/FileImporter@master] Fix constructor invocation and content model

https://gerrit.wikimedia.org/r/612417

gerritbot added a comment.Jul 14 2020, 10:48 AM

Change 612536 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] Add import/export round trip test.

https://gerrit.wikimedia.org/r/612536

gerritbot added a comment.Jul 14 2020, 11:35 AM

Change 612417 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Fix constructor invocation and content model

https://gerrit.wikimedia.org/r/612417

daniel mentioned this in rEFLI2cf3cf5faa03: Fix constructor invocation and content model.Jul 14 2020, 11:37 AM
ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.1; 2020-07-21).Jul 14 2020, 12:00 PM
gerritbot added a comment.Jul 14 2020, 2:27 PM

Change 612536 merged by jenkins-bot:
[mediawiki/core@master] Add import/export round trip test.

https://gerrit.wikimedia.org/r/612536

gerritbot added a comment.Jul 18 2020, 10:19 PM

Change 614327 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[mediawiki/core@master] Check for unknown slot.

https://gerrit.wikimedia.org/r/614327

gerritbot added a comment.Jul 20 2020, 12:34 PM

Change 586316 merged by jenkins-bot:
[mediawiki/core@master] MCR import

https://gerrit.wikimedia.org/r/586316

gerritbot added a comment.Jul 20 2020, 1:21 PM

Change 614044 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@REL1_35] Add import/export round trip test.

https://gerrit.wikimedia.org/r/614044

gerritbot added a comment.Jul 20 2020, 1:31 PM

Change 614044 merged by jenkins-bot:
[mediawiki/core@REL1_35] Add import/export round trip test.

https://gerrit.wikimedia.org/r/614044

gerritbot added a comment.Jul 20 2020, 1:50 PM

Change 614045 had a related patch set uploaded (by Cicalese; owner: Daniel Kinzler):
[mediawiki/core@REL1_35] MCR import

https://gerrit.wikimedia.org/r/614045

gerritbot added a comment.Jul 20 2020, 1:53 PM

Change 614766 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[mediawiki/extensions/FileImporter@REL1_35] Fix constructor invocation and content model

https://gerrit.wikimedia.org/r/614766

ReleaseTaggerBot added a project: MW-1.35-notes.Jul 20 2020, 2:00 PM
gerritbot added a comment.Jul 20 2020, 2:01 PM

Change 614045 merged by jenkins-bot:
[mediawiki/core@REL1_35] MCR import

https://gerrit.wikimedia.org/r/614045

gerritbot added a comment.Jul 20 2020, 2:08 PM

Change 614766 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@REL1_35] Fix constructor invocation and content model

https://gerrit.wikimedia.org/r/614766

CCicalese_WMF mentioned this in rEFLI622f0544b58a: Fix constructor invocation and content model.Jul 20 2020, 2:10 PM
gerritbot added a comment.Jul 20 2020, 2:51 PM

Change 614762 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[mediawiki/core@master] Remove backward compatibility code from ImportableOldRevisionImporter

https://gerrit.wikimedia.org/r/614762

gerritbot added a comment.Jul 20 2020, 2:54 PM

Change 614752 had a related patch set uploaded (by Cicalese; owner: Cicalese):
[mediawiki/extensions/FileImporter@master] Add SlotRoleRegistry to ImportableOldRevisionImporter constructor

https://gerrit.wikimedia.org/r/614752

gerritbot added a comment.Jul 20 2020, 3:20 PM

Change 614327 merged by jenkins-bot:
[mediawiki/core@master] Check for unknown slot.

https://gerrit.wikimedia.org/r/614327

gerritbot added a comment.Jul 20 2020, 3:45 PM

Change 614769 had a related patch set uploaded (by Daniel Kinzler; owner: Cicalese):
[mediawiki/core@REL1_35] Check for unknown slot.

https://gerrit.wikimedia.org/r/614769

• Nuria unsubscribed.Jul 20 2020, 3:47 PM
gerritbot added a comment.Jul 20 2020, 3:54 PM

Change 614769 merged by jenkins-bot:
[mediawiki/core@REL1_35] Check for unknown slot.

https://gerrit.wikimedia.org/r/614769

gerritbot added a comment.Jul 20 2020, 4:31 PM

Change 614752 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add SlotRoleRegistry to ImportableOldRevisionImporter constructor

https://gerrit.wikimedia.org/r/614752

CCicalese_WMF mentioned this in rEFLI4cb99d5dc603: Add SlotRoleRegistry to ImportableOldRevisionImporter constructor.Jul 20 2020, 4:32 PM
gerritbot added a comment.Jul 23 2020, 12:09 PM

Change 615444 had a related patch set uploaded (by Daniel Kinzler; owner: Cicalese):
[mediawiki/extensions/FileImporter@REL1_35] Add SlotRoleRegistry to ImportableOldRevisionImporter constructor

https://gerrit.wikimedia.org/r/615444

gerritbot added a comment.Jul 23 2020, 12:13 PM

Change 615444 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@REL1_35] Add SlotRoleRegistry to ImportableOldRevisionImporter constructor

https://gerrit.wikimedia.org/r/615444

daniel mentioned this in rEFLIff3d56e84c3f: Add SlotRoleRegistry to ImportableOldRevisionImporter constructor.Jul 23 2020, 12:16 PM
gerritbot added a comment.Jul 23 2020, 12:50 PM

Change 614762 merged by jenkins-bot:
[mediawiki/core@master] Remove backward compatibility code from ImportableOldRevisionImporter

https://gerrit.wikimedia.org/r/614762

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.2; 2020-07-28); removed MW-1.36-notes (1.36.0-wmf.1; 2020-07-21).Jul 23 2020, 1:00 PM
daniel mentioned this in T212428: includes/Revision/RevisionStore.php: Main slot of revision (number) not found in database!.Jul 23 2020, 1:28 PM
• eprodromou closed this task as Resolved.Sep 15 2020, 8:54 PM
• eprodromou subscribed.

Congratulations and good job!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits