Page MenuHomePhabricator
Create Task
Maniphest T183490

MCR schema migration stage 4: Migrate External Store URLs (wmf production)
Open, MediumPublic
Actions

Description

We should migrate data stored in the External Store away from the text table: The External Store URL that is contained in the text blob can be written to the cont_address field (possibly with a prefix, to be decided, see External Store Integration). Then the corresponding rows can be deleted from the text table.

Details

SubjectRepoBranchLines +/-
Use a BlobStore for storing var dumpsmediawiki/extensions/AbuseFiltermaster+24 -92
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT174043 Deploy Multi-Content Revisions
 StalledNoneT174045 DB schema migration for MCR
 OpenNoneT183490 MCR schema migration stage 4: Migrate External Store URLs (wmf production)
 ResolvedNoneT183487 MCR schema migration stage 3: stop using legacy fields
 ResolvedtstarlingT183488 MCR schema migration stage 2: populate new fields
 ResolvedAnomieT183489 MCR schema migration stage 1: Fix Legacy Archive Rows
 ResolvedAnomieT183486 MCR schema migration stage 0: create tables
 ResolvedAnomieT36925 [MCR] create maintenance script for Migration of text from archive table to text table
 ResolvedAnomieT184629 Run maintenance/migrateArchiveText.php on all wikis
 ResolvedAnomieT191307 Run maintenance/populateArchiveRevId.php on all wikis
 ResolvedAnomieT193180 Clean up archive rows with duplicate revision IDs
 ResolvedAnomieT202032 Duplicate ar_rev_id values in several wikis
 ResolvedAnomieT174030 Implement DB schema migration tools for MCR
 ResolveddanielT174028 Finalize database schema for MCR content meta-data
ResolvedAnomieT182678 [MCR] Script for populating empty ar_rev_id fields
 ResolvedAnomieT182682 [MCR] Script(s) for populating new tables (slots, content, content_models, slot_roles)
 Resolved• MarosteguiT190153 DROP unused 'slots' table (WAS: In the slots table, replace slot_inherited with slot_origin)
 ResolvedAnomieT196172 Test schema migration script
 ResolvedtstarlingT197816 Enable MCR migration stage "write both, read old" on live systems
 ResolvedAnomieT197817 Enable MCR migration stage "write both, read old" on testwiki
 ResolvedAnomieT198312 Set the WMF cluster to use the new MCR-only schema
 ResolveddanielT198558 Set testwiki to use the new MCR-only schema
 ResolveddanielT198559 Set Beta Cluster wikis to use the new MCR-only schema
 ResolvedAnomieT198311 Enable MCR migration stage "write both, read new" on commons beta.
 ResolvedAnomieT197818 Enable MCR migration stage "write both, read old" on commons beta.
 ResolveddanielT197619 Introduce SCHEMA_COMPAT_XXX constants to allow more fine grained control over the migration process
 ResolveddanielT198413 Allow multiple slots to be used while still writing to the old as well as the new schema
 ResolveddanielT236735 Allow WikiExporter to function with MCR-only schema
 ResolveddanielT198706 Make BackupDumper, WikiExporter and XmlDumpWriter compliant with the MCR revision retrival mechanism (main slot only)
 ResolvedBPirkleT203424 Replace the WikiExporter backup dump streaming mode with batched queries
 ResolvedNoneT174047 Hide deprecated/unused fields on toolforge replica [MCR]
 OpenLadsgroupT362566 Stop growth of text table by storing ES addresses in content table

Event Timeline

daniel triaged this task as Medium priority.Dec 21 2017, 6:43 PM
daniel created this task.
daniel added a subtask: T183487: MCR schema migration stage 3: stop using legacy fields.
Jdforrester-WMF subscribed.Jan 9 2018, 5:49 PM
CCicalese_WMF added a project: Multi-Content-Revisions.Jan 12 2018, 4:50 PM
Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Apr 23 2018, 7:28 AM
CCicalese_WMF moved this task from Inbox to MCR-SDC Storage Layer - phase 1 on the Multi-Content-Revisions board.May 8 2018, 1:16 PM
CCicalese_WMF edited projects, added Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1); removed Multi-Content-Revisions.
CCicalese_WMF moved this task from MCR-SDC Storage Layer - phase 1 to MCR-SDC File Caption Support - phase 2 on the Multi-Content-Revisions board.May 8 2018, 8:40 PM
CCicalese_WMF edited projects, added Multi-Content-Revisions (MCR-SDC File Caption Support - phase 2); removed Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1).
daniel edited projects, added Multi-Content-Revisions; removed Multi-Content-Revisions (MCR-SDC File Caption Support - phase 2), SDC General.Jun 27 2018, 5:39 PM

Dropping this from the SDC backlog

daniel renamed this task from MCR schema migration stage 4: Migrate External Store URLs to MCR schema migration stage 4: Migrate External Store URLs (wmf production).Jul 1 2018, 12:09 PM
CCicalese_WMF edited projects, added Multi-Content-Revisions (Deployment); removed Multi-Content-Revisions.Jul 4 2018, 2:20 AM
daniel mentioned this in T199121: RFC: Spec for representing multiple content objects per revision (MCR) in XML dumps.Aug 6 2018, 6:05 PM
daniel mentioned this in T183487: MCR schema migration stage 3: stop using legacy fields.Sep 14 2018, 8:32 AM
jcrespo subscribed.Oct 2 2018, 9:54 AM

Maybe related or blocker: T106363

I am interested on this happening at some point, I just saw lots of revision data (not metadata) stored on the text table and its impact on a quick database recovery.

CCicalese_WMF added projects: Platform Team Legacy (Later), Platform Engineering (MCR: Deployment).Oct 3 2018, 5:21 PM
CCicalese_WMF moved this task from MCR: Deployment to MCR on the Platform Engineering board.Oct 4 2018, 3:12 PM
CCicalese_WMF edited projects, added Platform Engineering (MCR); removed Platform Engineering (MCR: Deployment).
ArielGlenn subscribed.Oct 4 2018, 8:08 PM
daniel mentioned this in T220493: Xml stubs dumps are running 5 to 15x slower than previously.Jun 4 2019, 6:48 PM
WDoranWMF moved this task from Later to Mop Column on the Platform Team Legacy board.Jul 5 2019, 2:13 PM
WDoranWMF removed a project: Platform Team Legacy (Later).
WDoranWMF moved this task from MCR to mop on the Platform Engineering board.Jul 26 2019, 6:39 PM
WDoranWMF edited projects, added Platform Team Initiatives (MCR); removed Platform Engineering (MCR).
Catrope mentioned this in T106363: Migrate Flow content to new separate logical External Store in production.Aug 28 2019, 10:24 AM
CCicalese_WMF removed a subtask: T183487: MCR schema migration stage 3: stop using legacy fields.Aug 31 2019, 12:05 AM
CCicalese_WMF added a subtask: T183487: MCR schema migration stage 3: stop using legacy fields.Aug 31 2019, 12:27 AM
gerritbot added a comment.Sep 23 2019, 12:24 PM

Change 538599 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/AbuseFilter@master] Use a BlobStore for stroing var dumps.

https://gerrit.wikimedia.org/r/538599

gerritbot added a project: Patch-For-Review.Sep 23 2019, 12:24 PM
gerritbot added a comment.Oct 28 2019, 1:06 PM

Change 538599 abandoned by Daniel Kinzler:
Use a BlobStore for storing var dumps.

Reason:
see I22cf698c5be77506727cbd227c67e037a5d89b5c

https://gerrit.wikimedia.org/r/538599

Maintenance_bot removed a project: Patch-For-Review.Oct 28 2019, 1:11 PM
CCicalese_WMF closed subtask T183487: MCR schema migration stage 3: stop using legacy fields as Resolved.Nov 22 2019, 9:35 PM
Bugreporter mentioned this in T241312: Remove support for HistoryBlobCurStub.Dec 22 2019, 8:10 AM
gerritbot added a comment.Feb 29 2020, 5:04 PM

Change 538599 restored by Daimona Eaytoy:
Use a BlobStore for storing var dumps.

Reason:
This can be rebased on top of I4444cada720ab62d187f2dd0c4760697e465f2ff to remove any concern about back-compat stuff.

@daniel would you like to do that yourself, or should I revamp this patch and you/CPT will review?

https://gerrit.wikimedia.org/r/538599

daniel mentioned this in T261889: Use a BlobStore for storing var dumps.Sep 2 2020, 8:51 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:02 PM
gerritbot added a comment.Dec 15 2020, 2:58 AM

Change 538599 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Use a BlobStore for storing var dumps

https://gerrit.wikimedia.org/r/538599

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.25; 2021-01-05).Dec 15 2020, 3:00 AM
Izno added a subtask: T174047: Hide deprecated/unused fields on toolforge replica [MCR].Nov 8 2021, 10:04 PM
Ladsgroup mentioned this in T297633: <Tech Initiative> Improving Databases in MediaWiki.Jan 24 2022, 8:02 PM
Bugreporter mentioned this in T301039: Provide a dump of PDF/DjVU metadata.Feb 5 2022, 3:39 PM
Bugreporter mentioned this in T63111: Convert primary key integers and references thereto from int to bigint (unsigned).Apr 12 2022, 11:38 AM
Pppery closed subtask T174047: Hide deprecated/unused fields on toolforge replica [MCR] as Resolved.Apr 6 2023, 3:58 AM
Ladsgroup subscribed.Jun 13 2023, 10:34 AM

While doing the migration itself is a lot of work, making new revisions directly link to ES in content table should be rather easy and it would at least stop the table to grow even larger. Does anyone feel like doing that part?

Addshore unsubscribed.Jun 27 2023, 12:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL