Page MenuHomePhabricator
Create Task
Maniphest T198706

Make BackupDumper, WikiExporter and XmlDumpWriter compliant with the MCR revision retrival mechanism (main slot only)
Closed, ResolvedPublic
Actions

Description

For access to revision content, WikiExporter/XmlDumpWriter rely on joining the revision table to the text table, based on the rev_text_id field. This will completely stop working when leaving the write-both compatibility mode for MCR schema migration.

This task does not require support for extra slots to be added to dumps. It only requires the new revision retrieval interface to be used by the export/dump code.

Details

SubjectRepoBranchLines +/-
Make BackupDumper MCR compatible (main slot only)mediawiki/coremaster+308 -118
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenBUG REPORTNoneT12863 Links on commons upload summaries do not link to commons
 OpenNoneT5498 Image history is confusing
 StalledNoneT96384 Integrate file revisions with description page history
 OpenNoneT115858 Design improvements for mw.ForeignStructuredUpload.BookletLayout
 OpenNoneT115865 Insert image in content immediately after it's uploaded, skipping the "General settings" step
 DuplicateNoneT115864 Figure out if the description of the image can be used as the caption on-wiki
 OpenFeatureNoneT53032 When inserting an image, set its caption by default to be the Commons image description
 OpenFeatureNoneT39534 Wikimedia Commons should support searching by color
 DuplicateNoneT39535 Wikimedia Commons should support filtering by color
 ResolvedNoneT19503 Provide metadata support on Wikimedia Commons
 OpenNoneT122038 Moving/Deleting a video file does not move/delete subtitles files, nor does it inform the file mover of their existence
 OpenNoneT135221 Make TimedText content an integral part of the File page
 OpenFeatureNoneT2167 Use a dedicated interface for adding meta-data like interwiki links, rather than wikitext
 OpenNoneT132072 Integrate page meta-data as a new content model revision slot for consistency and atomicity
 ResolvedNoneT51662 VisualEditor: Use Multimedia/Wikidata's proposed rich structured meta-data in the image insertion dialog
 ResolvedNoneT68108 [Epic] Store media information for files on Wikimedia Commons as structured data
 OpenFeatureNoneT14963 allow per-page exceptions to spam blacklist
 OpenNoneT203157 Make the spam whitelist its own slot
 OpenFeatureNoneT56140 Move TemplateData to its own JSON-content namespace and associate with Template-namespace, or to its own TemplateData content model and revision slot
 StalledNoneT174043 Deploy Multi-Content Revisions
 OpenNoneT174022 Implement multi-content revisions
 InvalidNoneT198492 Create a maintenance script to drop rev_text_id and ar_text_id from the database.
 StalledNoneT174045 DB schema migration for MCR
 ResolvedAnomieT174030 Implement DB schema migration tools for MCR
 ResolveddanielT174028 Finalize database schema for MCR content meta-data
ResolvedAnomieT182678 [MCR] Script for populating empty ar_rev_id fields
 ResolvedAnomieT182682 [MCR] Script(s) for populating new tables (slots, content, content_models, slot_roles)
 OpenZabeT183490 MCR schema migration stage 4: Migrate External Store URLs (wmf production)
 Resolved• BstormT184446 Configure Toolforge replica views and dumps for the new MCR tables
 ResolvedNoneT231668 MCR schema migration
 ResolvedZabeT231674 [Epic] Remove support for writing to the pre-MCR schema
 ResolveddanielT198557 Remove the ability to write pre-MCR fields, limit the ability to read pre-MCR fields to migration scripts
 ResolveddanielT231673 Make SCHEMA_COMPAT_NEW the default MCR migration schema
 Resolved• ClarakosiT214308 Force usage of MCR aware database schema
 ResolvedNoneT231672 Stop writing to pre-MCR fields in WMF production
 ResolvedNoneT183487 MCR schema migration stage 3: stop using legacy fields
 ResolvedAnomieT198312 Set the WMF cluster to use the new MCR-only schema
 ResolvedNoneT231671 [EPIC] Ensure all direct or indirect access to pre-MCR fields is gated with the MCR migration stage and emits a warning if any pre-MCR schema fields are accessed
 ResolvedCCicalese_WMFT231689 Can get bulk access from code that has been using the 'text' for getQueryInfo()
Resolved• PcheloloT230834 Create batch access interface for BlobStore
 Resolved• PcheloloT228988 Create batch access interface for page content
 Resolved• PcheloloT233173 Make RevisionStore::newRevisionsFromBatch use BlobStore::getBlobBatch
 ResolvedCCicalese_WMFT231693 No direct or indirect access to pre-MCR fields is ungated
 ResolveddanielT198341 Remove all references to the rev_text_id and ar_text_id fields
 Resolved• PcheloloT198342 Remove all usages of the 'text' flag in calls to Revision::getQueryInfo() and RevisionStore::getQueryInfo().
 ResolvedNoneT231699 A warning is emitted if any pre-MCR fields are accessed
 ResolvedNoneT200918 Make sure code that accesses legacy pre-MCR fields triggers warnings before switching off WRITE_OLD (compat) mode.
 ResolveddanielT174031 MCR: Include all slots in XML dumps
 ResolveddanielT198706 Make BackupDumper, WikiExporter and XmlDumpWriter compliant with the MCR revision retrival mechanism (main slot only)
 ResolvedBPirkleT203424 Replace the WikiExporter backup dump streaming mode with batched queries

Event Timeline

daniel triaged this task as Medium priority.Jul 3 2018, 12:46 PM
daniel created this task.
daniel added parent tasks: T198312: Set the WMF cluster to use the new MCR-only schema, T198557: Remove the ability to write pre-MCR fields, limit the ability to read pre-MCR fields to migration scripts.Jul 3 2018, 12:48 PM
gerritbot subscribed.Jul 3 2018, 12:52 PM

Change 443608 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] Make BackupDumper MCR compatible (main slot only)

https://gerrit.wikimedia.org/r/443608

gerritbot added a project: Patch-For-Review.Jul 3 2018, 12:52 PM
daniel added a parent task: T198342: Remove all usages of the 'text' flag in calls to Revision::getQueryInfo() and RevisionStore::getQueryInfo()..Jul 3 2018, 12:53 PM
daniel updated the task description. (Show Details)Jul 9 2018, 3:49 PM
daniel removed a parent task: T174031: MCR: Include all slots in XML dumps.
daniel renamed this task from Make BackupDumper, WikiExporter and XmlDumpWriter compliant with the MCR revision retrival mechanism. to Make BackupDumper, WikiExporter and XmlDumpWriter compliant with the MCR revision retrival mechanism (main slot only).Jul 9 2018, 3:52 PM
daniel added a parent task: T174031: MCR: Include all slots in XML dumps.
daniel updated the task description. (Show Details)
daniel edited projects, added Multi-Content-Revisions; removed Patch-For-Review, Multi-Content-Revisions (MCR-SDC File Caption Support - phase 2).Jul 9 2018, 4:00 PM
This comment was removed by daniel.
daniel edited projects, added Patch-For-Review, Multi-Content-Revisions (MCR-SDC File Caption Support - phase 2); removed Multi-Content-Revisions.Jul 9 2018, 4:02 PM
daniel claimed this task.Jul 9 2018, 4:12 PM
daniel moved this task from Backlog to In Progress on the Multi-Content-Revisions (MCR-SDC File Caption Support - phase 2) board.
daniel moved this task from In Progress to Needs Review on the Multi-Content-Revisions (MCR-SDC File Caption Support - phase 2) board.Jul 9 2018, 5:28 PM
ArielGlenn subscribed.Jul 19 2018, 9:28 AM
daniel edited projects, added Multi-Content-Revisions; removed Multi-Content-Revisions (MCR-SDC File Caption Support - phase 2).Jul 31 2018, 12:13 PM
daniel added a subtask: T203424: Replace the WikiExporter backup dump streaming mode with batched queries.Sep 7 2018, 12:19 PM
Addshore moved this task from incoming to monitoring on the Wikidata board.Sep 17 2018, 8:20 AM
daniel moved this task from Inbox to In Progress on the Multi-Content-Revisions board.Sep 17 2018, 1:09 PM
Jdforrester-WMF edited projects, added Structured Data Engineering; removed SDC General.Sep 27 2018, 10:12 PM
Jdforrester-WMF moved this task from To Do to Monitoring on the Structured Data Engineering board.Sep 27 2018, 10:13 PM
BPirkle closed subtask T203424: Replace the WikiExporter backup dump streaming mode with batched queries as Resolved.Oct 3 2018, 5:14 PM
CCicalese_WMF added projects: Platform Team Legacy (Designing), Platform Engineering (MCR).Oct 3 2018, 5:14 PM
CCicalese_WMF moved this task from Inbox to Doing on the Platform Team Legacy (Designing) board.Oct 3 2018, 7:20 PM
CCicalese_WMF moved this task from In Progress to Tech Debt on the Multi-Content-Revisions board.Oct 4 2018, 2:58 PM
CCicalese_WMF edited projects, added Multi-Content-Revisions (Tech Debt); removed Multi-Content-Revisions.
Jdforrester-WMF mentioned this in T206884: Provide appropriate dumps of Commons including the structured data.Oct 12 2018, 9:39 PM
CCicalese_WMF edited projects, added Platform Team Workboards (Doing); removed Platform Team Legacy (Designing).Oct 14 2018, 3:55 PM
CCicalese_WMF moved this task from Doing to Waiting for Review on the Platform Team Workboards board.Nov 13 2018, 3:41 PM
CCicalese_WMF edited projects, added Platform Team Workboards (Waiting for Review); removed Platform Team Workboards (Doing).
daniel added a parent task: T214308: Force usage of MCR aware database schema.Jan 21 2019, 2:42 PM
daniel moved this task from Waiting for Review to Doing on the Platform Team Workboards board.Feb 19 2019, 3:57 PM
daniel edited projects, added Platform Team Workboards (Doing); removed Platform Team Workboards (Waiting for Review).
daniel mentioned this in T217607: MediaWikiTestCase::copyTestData does not work on Postgres and SQLite.Mar 4 2019, 9:49 PM
greg added a project: Multimedia.Mar 7 2019, 10:59 PM
• Ramsey-WMF removed a project: Multimedia.Mar 8 2019, 4:20 PM
Restricted Application added a project: Multimedia. · View Herald TranscriptMar 8 2019, 4:20 PM
• Ramsey-WMF removed a project: Multimedia.Mar 8 2019, 4:21 PM
BPirkle mentioned this in T198341: Remove all references to the rev_text_id and ar_text_id fields.Mar 13 2019, 3:56 PM
daniel edited projects, added Platform Team Workboards (Waiting for Review); removed Platform Team Workboards (Doing).Mar 14 2019, 7:56 PM
gerritbot added a comment.Mar 20 2019, 2:15 AM

Change 443608 merged by jenkins-bot:
[mediawiki/core@master] Make BackupDumper MCR compatible (main slot only)

https://gerrit.wikimedia.org/r/443608

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.23; 2019-03-26).Mar 20 2019, 3:01 AM
daniel closed this task as Resolved.May 15 2019, 10:03 PM
• WDoranWMF moved this task from Waiting for Review to Ready on the Platform Team Workboards board.Jul 5 2019, 3:27 PM
• WDoranWMF edited projects, added Platform Team Workboards (Done with CPT); removed Platform Team Workboards (Waiting for Review).
CCicalese_WMF edited projects, added Core Platform Team Initiatives (MCR); removed Platform Engineering (MCR).Jul 30 2019, 7:49 PM
daniel mentioned this in T236735: Allow WikiExporter to function with MCR-only schema.Oct 28 2019, 8:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits