Page MenuHomePhabricator
Create Task
Maniphest T198341

Remove all references to the rev_text_id and ar_text_id fields
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Any code that accesses rev_text_id and ar_text_id fields is unable to make use of MCR. In order to enable for MCR support, all references to these fields have to be removed from the code. They should be replaced with RevisionStore::getQueryInfo(), RevisionStore::newRevisionFromRow(), and RevisionRecord::getContent().

As a corollary, all references to text.old_id should be removed as well. If rev_text_id and ar_text_id are no longer there, there is no need to reference text.old_id (except in SqlBlobStore). A search for uses of the _text_id suffix in may be helpful to find code that needs fixing.

DOD: This task should be considered done when any code referencing rev_text_id or ar_text_id has either been removed or is gated on the MCR schema migration stage, in core and all extensions deployed by Wikimedia, as well as extensions bundled with MediaWiki releases. Other extensions and external tools may be left to be updated later.

Extensions to fix:

NOTE: fixes for this will probably coincide with fixes for T198343: Replace all calls to Revision::getRevisionText() and T198342: Remove all usages of the 'text' flag in calls to Revision::getQueryInfo() and RevisionStore::getQueryInfo()..

Details

SubjectRepoBranchLines +/-
Remove references to db fields rev_text_id and ar_text_idmediawiki/extensions/DeletePagesForGoodmaster+137 -17
Remove references to db fields rev_text_id and ar_text_idmediawiki/extensions/DeletePagesForGoodmaster+117 -17
Use a BlobStore for storing var dumpsmediawiki/extensions/AbuseFiltermaster+24 -92
Revision::getRevisionText(): support old schemasmediawiki/coremaster+197 -69
MCR WIP Remove references to rev_text_id and ar_text_idmediawiki/coremaster+23 -15
Only use the revision.rev_text_id if it exists.mediawiki/extensions/Duplicatormaster+4 -2
Remove references to db field ar_text_idmediawiki/coremaster+10 -43
Remove references to field rev_text_idmediawiki/coremaster+95 -53
Remove references to field rev_text_idmediawiki/coremaster+28 -8
Remove many references to db fields being retired as part of MCR Schema Migrationmediawiki/coremaster+58 -35
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 DeclineddchenT118706 Conduct heuristic evaluation of image upload and insert flow in VisualEditor
 OpenNoneT115858 Design improvements for mw.ForeignStructuredUpload.BookletLayout
 OpenNoneT115865 Insert image in content immediately after it's uploaded, skipping the "General settings" step
 DuplicateNoneT115864 Figure out if the description of the image can be used as the caption on-wiki
 OpenFeatureNoneT53032 When inserting an image, set its caption by default to be the Commons image description
 OpenBUG REPORTNoneT12863 Links on commons upload summaries do not link to commons
 OpenNoneT5498 Image history is confusing
 StalledNoneT96384 Integrate file revisions with description page history
 OpenFeatureNoneT39534 Wikimedia Commons should support searching by color
 DuplicateNoneT39535 Wikimedia Commons should support filtering by color
 ResolvedNoneT19503 Provide metadata support on Wikimedia Commons
 OpenNoneT122038 Moving/Deleting a video file does not move/delete subtitles files, nor does it inform the file mover of their existence
 OpenNoneT135221 Make TimedText content an integral part of the File page
 OpenFeatureNoneT2167 Use a dedicated interface for adding meta-data like interwiki links, rather than wikitext
 OpenNoneT132072 Integrate page meta-data as a new content model revision slot for consistency and atomicity
 ResolvedNoneT51662 VisualEditor: Use Multimedia/Wikidata's proposed rich structured meta-data in the image insertion dialog
 ResolvedNoneT68108 [Epic] Store media information for files on Wikimedia Commons as structured data
 OpenFeatureNoneT14963 allow per-page exceptions to spam blacklist
 OpenNoneT203157 Make the spam whitelist its own slot
 OpenFeatureNoneT56140 Move TemplateData to its own JSON-content namespace and associate with Template-namespace, or to its own TemplateData content model and revision slot
 StalledNoneT174043 Deploy Multi-Content Revisions
 OpenNoneT174022 Implement multi-content revisions
 InvalidNoneT198492 Create a maintenance script to drop rev_text_id and ar_text_id from the database.
 ResolvedNoneT231668 MCR schema migration
 ResolvedZabeT231674 [Epic] Remove support for writing to the pre-MCR schema
 ResolveddanielT198557 Remove the ability to write pre-MCR fields, limit the ability to read pre-MCR fields to migration scripts
 ResolvedCCicalese_WMFT242764 Test schema update from old versions to MCR-only schema
 ResolveddanielT231673 Make SCHEMA_COMPAT_NEW the default MCR migration schema
 ResolvedNoneT231672 Stop writing to pre-MCR fields in WMF production
 ResolvedNoneT231671 [EPIC] Ensure all direct or indirect access to pre-MCR fields is gated with the MCR migration stage and emits a warning if any pre-MCR schema fields are accessed
 ResolvedCCicalese_WMFT231693 No direct or indirect access to pre-MCR fields is ungated
 ResolveddanielT198341 Remove all references to the rev_text_id and ar_text_id fields
 Resolved• PcheloloT198342 Remove all usages of the 'text' flag in calls to Revision::getQueryInfo() and RevisionStore::getQueryInfo().
 ResolveddanielT198706 Make BackupDumper, WikiExporter and XmlDumpWriter compliant with the MCR revision retrival mechanism (main slot only)
 ResolvedBPirkleT203424 Replace the WikiExporter backup dump streaming mode with batched queries
 Resolved• PcheloloT233356 Flow: remove dependency on pre-MCR database schema
 Resolved• ClarakosiT233357 ReplaceText: remove dependency on pre-MCR database schema
 ResolveddanielT233358 SpamBlacklist: remove dependency on pre-MCR database schema
 Resolved• PcheloloT198343 Replace all calls to Revision::getRevisionText()
 Resolved• PcheloloT228675 Remove direct access to the text table from the Translate extension.
 Resolved• PcheloloT228988 Create batch access interface for page content
 Resolved• PcheloloT233173 Make RevisionStore::newRevisionsFromBatch use BlobStore::getBlobBatch
 Resolved• PcheloloT230834 Create batch access interface for BlobStore
 ResolveddanielT234034 Create low level batch access interface for page content
 ResolveddanielT233350 ActiveAbstract: remove dependency on pre-MCR database schema
 Resolved• holger.knustT233353 EntitySchema: remove dependency on pre-MCR database schema
 Resolved• PcheloloT233355 WikimediaMaintenance: remove dependency on pre-MCR database schema
 ResolveddanielT238958 Remove pre-MCR schema fields during update
 ResolvedNoneT174047 Hide deprecated/unused fields on toolforge replica [MCR]

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Anomie added a comment.Apr 25 2019, 3:54 PM
In T198341#5137816, @Anomie wrote:

For that matter, I should run a query on the Analytics data to see whether we can just remove ApiQueryDeletedrevs entirely yet.

T221869: Remove deprecated ApiQueryDeletedRevs

Possibly still worth removing the ar_text_id uses to close out this task though.

gerritbot added a comment.Apr 25 2019, 11:39 PM

Change 506573 had a related patch set uploaded (by BPirkle; owner: BPirkle):
[mediawiki/extensions/DeletePagesForGood@master] Remove references to db fields rev_text_id and ar_text_id

https://gerrit.wikimedia.org/r/506573

CCicalese_WMF moved this task from Waiting for Review to Doing on the Platform Team Workboards board.May 13 2019, 3:03 PM
CCicalese_WMF edited projects, added Platform Team Workboards (Doing); removed Platform Team Workboards (Waiting for Review).
CCicalese_WMF edited projects, added Platform Team Workboards (Team 3); removed Platform Team Workboards (Doing).Jul 1 2019, 2:51 PM
CCicalese_WMF moved this task from Backlog to Doing on the Platform Team Workboards (Team 3) board.
gerritbot added a comment.Jul 10 2019, 3:01 PM

Change 506325 merged by jenkins-bot:
[mediawiki/core@master] Remove references to db field ar_text_id

https://gerrit.wikimedia.org/r/506325

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.14; 2019-07-16); removed MW-1.34-notes (1.34.0-wmf.3; 2019-04-30).Jul 10 2019, 4:00 PM
BPirkle mentioned this in T227835: Add notion of deletion to BlobStore.Jul 11 2019, 11:49 PM
BPirkle mentioned this in T227836: Create maintenance script to delete orphaned content.
GTrang added a subtask: T220884: Update the DeletePagesForGood extension for MCR.Jul 12 2019, 12:37 AM
gerritbot added a comment.Jul 12 2019, 2:24 AM

Change 522228 had a related patch set uploaded (by BPirkle; owner: BPirkle):
[mediawiki/extensions/Duplicator@master] Only use the revision.rev_text_id if it exists.

https://gerrit.wikimedia.org/r/522228

BPirkle mentioned this in T227840: Update the Duplicator extension for MCR.Jul 12 2019, 2:32 AM
BPirkle added a comment.Jul 12 2019, 2:39 AM

I've uploaded patches for all issues identified in this task. In an effort to keep this task focused on ar_text_id and rev_text_id, I created several related tasks for things that deserve attention and which are related to MCR, but not directly to rev_text_id or ar_text_id:

T220884 (additional DeletePagesForGood extension updates)
T227836 (maintenance script related to DeletePagesForGood, but possibly also generically useful)
T227840 (additional Duplicator extension updates)

While it may be slightly disingenuous to push the additional work to other tasks and declare this task done, I think it does make sense to move the extension work that is not part of the MCR Schema migration critical path to more focused tasks. Focused tasks are good practice, and extension-specific tasks are more likely to be noticed by people interested in those specific extensions.

daniel updated the task description. (Show Details)Jul 17 2019, 10:56 AM
daniel removed a subtask: T220884: Update the DeletePagesForGood extension for MCR.
daniel mentioned this in T220884: Update the DeletePagesForGood extension for MCR.
WDoranWMF edited projects, added Platform Team Workboards (Architecture Review Workboard); removed Platform Team Workboards (Team 3).Jul 17 2019, 4:21 PM
gerritbot added a comment.Jul 19 2019, 4:59 PM

Change 524560 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] Revision::getRevisionText(): support old schemas

https://gerrit.wikimedia.org/r/524560

daniel added a comment.Jul 19 2019, 8:10 PM

It seems like our search for usages in extensions failed to take into account usages in joins. There is quite a bit of code that does this. Critically, the Translate extension heavily depends on rev_text_id in joins.

Here's the code search:
https://codesearch.wmflabs.org/extensions/?q=rev_text_id%7Car_text_id&i=nope&files=&repos=

gerritbot added a comment.Jul 19 2019, 8:48 PM

Change 524593 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Translate@master] Remove direct access to text table

https://gerrit.wikimedia.org/r/524593

Addshore unsubscribed.Jul 20 2019, 3:07 AM
daniel updated the task description. (Show Details)Jul 22 2019, 2:52 PM
daniel updated the task description. (Show Details)Jul 22 2019, 2:56 PM
daniel mentioned this in T198343: Replace all calls to Revision::getRevisionText().Jul 22 2019, 3:01 PM
daniel updated the task description. (Show Details)
daniel updated the task description. (Show Details)
daniel mentioned this in T228675: Remove direct access to the text table from the Translate extension..Jul 22 2019, 3:11 PM
daniel added a subtask: T228675: Remove direct access to the text table from the Translate extension..
daniel updated the task description. (Show Details)Jul 22 2019, 3:19 PM
• Fjalapeno moved this task from Backlog to Engineering Tasks Ready for Estimation on the Platform Team Workboards (Architecture Review Workboard) board.Jul 23 2019, 1:28 PM
• Fjalapeno moved this task from Engineering Tasks Ready for Estimation to Backlog on the Platform Team Workboards (Architecture Review Workboard) board.
• Fjalapeno removed BPirkle as the assignee of this task.Jul 23 2019, 6:23 PM
• Fjalapeno added a subscriber: BPirkle.
WDoranWMF edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Team Workboards (Architecture Review Workboard).Jul 23 2019, 6:24 PM
• Fjalapeno edited projects, added Platform Engineering; removed Platform Engineering (MCR).Jul 23 2019, 6:24 PM
WDoranWMF raised the priority of this task from Medium to High.Jul 23 2019, 6:24 PM
WDoranWMF moved this task from Inbox to Later on the Platform Team Workboards (Clinic Duty Team) board.
WDoranWMF moved this task from Later to Ready (WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.
WDoranWMF removed a project: Platform Engineering.
CCicalese_WMF added a project: Platform Engineering (MCR).Jul 23 2019, 9:20 PM
WDoranWMF lowered the priority of this task from High to Medium.Jul 24 2019, 9:46 PM
WDoranWMF moved this task from Ready (WIP:5) to Later on the Platform Team Workboards (Clinic Duty Team) board.
WDoranWMF moved this task from MCR to mop on the Platform Engineering board.Jul 26 2019, 6:38 PM
WDoranWMF edited projects, added Core Platform Team Initiatives (MCR); removed Platform Engineering (MCR).
CCicalese_WMF edited projects, added Core Platform Team Initiatives (MCR Schema Migration); removed Core Platform Team Initiatives (MCR).Aug 1 2019, 3:34 AM
WDoranWMF moved this task from Later to Inbox on the Platform Team Workboards (Clinic Duty Team) board.Aug 6 2019, 3:20 PM
Aklapper added a project: Structured-Data-Backlog.Aug 10 2019, 11:55 PM
• Ramsey-WMF moved this task from Triage to Tracking on the Structured-Data-Backlog board.Aug 13 2019, 7:27 PM
CCicalese_WMF added a parent task: T231693: No direct or indirect access to pre-MCR fields is ungated.Aug 30 2019, 10:32 PM
CCicalese_WMF removed parent tasks: T184615: Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields from the revision table, T198557: Remove the ability to write pre-MCR fields, limit the ability to read pre-MCR fields to migration scripts, T198492: Create a maintenance script to drop rev_text_id and ar_text_id from the database., T183487: MCR schema migration stage 3: stop using legacy fields, T198312: Set the WMF cluster to use the new MCR-only schema.Aug 30 2019, 10:51 PM
CCicalese_WMF edited projects, added Platform Team Workboards (Architecture Review Workboard); removed Platform Team Workboards (Clinic Duty Team).Sep 17 2019, 4:05 PM
WDoranWMF moved this task from Backlog to Next Sprint on the Platform Team Workboards (Architecture Review Workboard) board.Sep 17 2019, 4:08 PM
WDoranWMF moved this task from Next Sprint to Engineering Tasks Ready for Estimation on the Platform Team Workboards (Architecture Review Workboard) board.
WDoranWMF moved this task from Engineering Tasks Ready for Estimation to Blocked on the Platform Team Workboards (Architecture Review Workboard) board.Sep 17 2019, 4:10 PM
WDoranWMF moved this task from Blocked to Engineering Tasks Ready for Estimation on the Platform Team Workboards (Architecture Review Workboard) board.Sep 17 2019, 4:12 PM
WDoranWMF changed the point value for this task from 1 to 5.Sep 17 2019, 4:28 PM
WDoranWMF moved this task from Engineering Tasks Ready for Estimation to Ready on the Platform Team Workboards (Architecture Review Workboard) board.
WDoranWMF moved this task from Ready to Blocked on the Platform Team Workboards (Architecture Review Workboard) board.Sep 17 2019, 4:54 PM
WDoranWMF moved this task from Blocked to Ready on the Platform Team Workboards (Architecture Review Workboard) board.
• holger.knust claimed this task.Sep 18 2019, 4:23 PM
• holger.knust moved this task from Ready to Doing on the Platform Team Workboards (Architecture Review Workboard) board.
daniel updated the task description. (Show Details)Sep 19 2019, 6:28 PM
daniel removed a subtask: T233350: ActiveAbstract: remove dependency on pre-MCR database schema.Sep 19 2019, 7:41 PM
daniel moved this task from Doing to Blocked on the Platform Team Workboards (Architecture Review Workboard) board.Sep 19 2019, 8:07 PM
daniel updated the task description. (Show Details)Sep 19 2019, 9:06 PM
gerritbot added a comment.Sep 19 2019, 9:10 PM

Change 538113 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/WikimediaMaintenance@master] Remove obsolete scripts that use rev_text_id.

https://gerrit.wikimedia.org/r/538113

• Pchelolo closed subtask T233356: Flow: remove dependency on pre-MCR database schema as Resolved.Sep 19 2019, 10:29 PM
daniel removed • holger.knust as the assignee of this task.Sep 20 2019, 12:12 PM
daniel added a subscriber: • holger.knust.
gerritbot added a comment.Sep 23 2019, 12:24 PM

Change 538599 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/AbuseFilter@master] Use a BlobStore for stroing var dumps.

https://gerrit.wikimedia.org/r/538599

• Clarakosi closed subtask T233357: ReplaceText: remove dependency on pre-MCR database schema as Resolved.Sep 23 2019, 3:05 PM
daniel closed subtask T233355: WikimediaMaintenance: remove dependency on pre-MCR database schema as Resolved.Sep 23 2019, 4:07 PM
• Pchelolo closed subtask T233353: EntitySchema: remove dependency on pre-MCR database schema as Resolved.Sep 27 2019, 5:31 PM
• Pchelolo closed subtask T228675: Remove direct access to the text table from the Translate extension. as Resolved.Oct 7 2019, 7:07 PM
• Pchelolo closed subtask T198343: Replace all calls to Revision::getRevisionText() as Resolved.
• Pchelolo closed subtask T198342: Remove all usages of the 'text' flag in calls to Revision::getQueryInfo() and RevisionStore::getQueryInfo(). as Resolved.Oct 7 2019, 7:15 PM
CCicalese_WMF updated the task description. (Show Details)Oct 21 2019, 6:18 PM
gerritbot added a comment.Oct 28 2019, 1:06 PM

Change 538599 abandoned by Daniel Kinzler:
Use a BlobStore for storing var dumps.

Reason:
see I22cf698c5be77506727cbd227c67e037a5d89b5c

https://gerrit.wikimedia.org/r/538599

daniel mentioned this in T236735: Allow WikiExporter to function with MCR-only schema.Oct 28 2019, 8:04 PM
gerritbot added a comment.Oct 28 2019, 8:38 PM

Change 442358 abandoned by Daniel Kinzler:
MCR WIP Remove references to rev_text_id and ar_text_id

Reason:
Obsolete,see I5ea972bb07ca1cfb3a2ad8ef120aef77e460745c

https://gerrit.wikimedia.org/r/442358

gerritbot added a comment.Oct 28 2019, 8:39 PM

Change 524560 abandoned by Daniel Kinzler:
Revision::getRevisionText(): support old schemas

Reason:
per Anomie's comment

https://gerrit.wikimedia.org/r/524560

daniel closed this task as Resolved.Oct 28 2019, 8:43 PM
daniel claimed this task.

All subtasks are resolved. I just went over the remaining open patches for this task and abandoned a few obsolete ones. There are only two patches remaining open, one for the Duplicator extension and one for the DeletePagesForGood extension. Neither of these is supported by WMF (deployed or bundled).

So, it seems we are done here. Closing.

daniel moved this task from Blocked to Done on the Platform Team Workboards (Architecture Review Workboard) board.Oct 28 2019, 8:43 PM
gerritbot added a comment.Feb 29 2020, 5:04 PM

Change 538599 restored by Daimona Eaytoy:
Use a BlobStore for storing var dumps.

Reason:
This can be rebased on top of I4444cada720ab62d187f2dd0c4760697e465f2ff to remove any concern about back-compat stuff.

@daniel would you like to do that yourself, or should I revamp this patch and you/CPT will review?

https://gerrit.wikimedia.org/r/538599

gerritbot added a comment.Mar 26 2020, 4:26 PM

Change 506573 abandoned by BPirkle:
Remove references to db fields rev_text_id and ar_text_id

Reason:
Extremely outdated

https://gerrit.wikimedia.org/r/506573

daniel mentioned this in T261889: Use a BlobStore for storing var dumps.Sep 2 2020, 8:51 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:41 PM
gerritbot added a comment.Dec 15 2020, 2:58 AM

Change 538599 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Use a BlobStore for storing var dumps

https://gerrit.wikimedia.org/r/538599

gerritbot added a comment.Sep 18 2023, 5:31 PM

Change 958530 had a related patch set uploaded (by Seb35; author: BPirkle):

[mediawiki/extensions/DeletePagesForGood@master] Remove references to db fields rev_text_id and ar_text_id

https://gerrit.wikimedia.org/r/958530

gerritbot added a comment.Sep 18 2023, 7:56 PM

Change 958530 merged by jenkins-bot:

[mediawiki/extensions/DeletePagesForGood@master] Remove references to db fields rev_text_id and ar_text_id

https://gerrit.wikimedia.org/r/958530

Seb35 mentioned this in rEDPG65d4c14590f8: Remove references to db fields rev_text_id and ar_text_id.Sep 18 2023, 7:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits