Page MenuHomePhabricator
Create Task
Maniphest T215466

Remove revision_comment_temp and revision_actor_temp
Open, Needs TriagePublic
Actions

Description

Due to expected difficulty in altering the revision table (see T161671: Compacting the revision table), the actor and comment migrations created tables named revision_comment_temp and revision_actor_temp to relate the rev_id and the comment_id/actor_id, with the plan being to eventually alter revision to contain rev_comment_id and rev_actor columns and drop those relation tables. This task tracks that "eventually".

  • 0. Wait for the main actor and comment migration to be done (except for DBA work).
    • 0.1. Comment migration, through steps 7.1 and 7.2 (T166733)
    • 0.2. Actor migration, through steps 7.1 and 7.2 (T188327)
  • 1. Alter revision to drop rev_comment, rev_user, and rev_user_text and add rev_comment_id and rev_actor. (T161671)
    • 1.1. Write and merge the MediaWiki patch.
    • 1.2. Have DBAs perform the change.
  • 2. Introduce a migration from using revision_comment_temp and revision_actor_temp (old) to rev_comment_id and rev_actor (new).
    • 2.1. Write and merge the MediaWiki patch, assuming it wasn't included in 1.1.
    • 2.2. Check for extensions assuming the temp tables exist.
  • 3. Set migration stage to write-both/read-old.
  • 4. Run the maintenance script to backfill rev_comment_id and rev_actor.
  • 5. Set migration stage to write-both/read-new.
    • 5.1. Announce changes to wikitech-l and cloud.
    • 5.2. Update WMCS views to read the new schema.
  • 6. Set migration stage to write-new/read-new.
  • 7. Remove old tables and code.

Details

ProjectBranchLines +/-Subject
mediawiki/coreREL1_39+25 -65installer: Split drop action out of the SQL patch for actor migration
operations/mediawiki-configmaster+0 -8Stop setting $wgActorTableSchemaMigrationStage
mediawiki/coreREL1_39+1 -116MigrateActors: Write to revsion table (Follow-up 24115a8)
operations/softwaremaster+0 -1tables_to_check: drop revision_actor_temp
mediawiki/coremaster+1 -20Remove references to revision_actor_temp, aliased as temp_rev_user
mediawiki/coremaster+49 -533Start clean up of revision_actor_temp table
mediawiki/coremaster+273 -157Alter `revision` for actor, comment, and MCR
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Bstorm added a subscriber: Bstorm.Jun 5 2019, 9:06 PM
CCicalese_WMF added a parent task: T227047: Complete actor table and comment table migration.Jul 1 2019, 11:48 PM
WDoranWMF removed a project: Platform Team Legacy (Epic).Jul 5 2019, 7:31 PM
WDoranWMF edited projects, added Platform Engineering (Revision Storage Schema Improvements); removed Platform Engineering (Needs Cleaning - Code Health (TEC13)).Jul 9 2019, 7:02 PM
WDoranWMF moved this task from Revision Storage Schema Improvements to mop on the Platform Engineering board.Jul 26 2019, 7:11 PM
WDoranWMF edited projects, added Platform Team Initiatives (Revision Storage Schema Improvements); removed Platform Engineering (Revision Storage Schema Improvements).
Anomie updated the task description. (Show Details)Sep 17 2019, 5:43 PM
Anomie added a comment.Sep 17 2019, 5:48 PM

Status: Blocked on T184615: Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields from the revision table also being ready to go. The intention is to combine all the alters of revision into one task for the DBAs.

Anomie mentioned this in T184615: Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields from the revision table.Sep 17 2019, 5:49 PM
Iflorez mentioned this in T234560: identify a reliable user field for use with the revision, page...maybe also the actor table.Oct 3 2019, 7:23 PM
Iflorez added a subscriber: Iflorez.Oct 3 2019, 8:25 PM
Nuria added a project: Analytics.Oct 21 2019, 7:40 PM
Nuria added a subscriber: Nuria.

This is going to affect Analytics mediawiki reconstruction, tagging

JAllemandou added a subscriber: JAllemandou.Oct 22 2019, 8:01 PM
Nuria added a subscriber: Milimetric.Oct 24 2019, 4:49 PM

ping @Milimetric

Nuria moved this task from Incoming to Radar on the Analytics board.Oct 24 2019, 5:16 PM
JAllemandou added a comment.Oct 28 2019, 10:36 AM

@Nuria : I don't think we use the tables mentioned here. Labs might use them in the background, but views are already maintained with 'the new' schema (@Milimetric can you confim?)

Milimetric added a comment.Nov 19 2019, 3:16 AM

I'm sorry I thought I confirmed this - yes, we were blissfully ignorant on top of the cloud db views. Our other private sqoops don't use these tables either.

gerritbot added a comment.Nov 21 2019, 9:30 PM

Change 552339 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@master] Alter revision for actor, comment, and MCR

https://gerrit.wikimedia.org/r/552339

gerritbot added a project: Patch-For-Review.Nov 21 2019, 9:30 PM
Anomie mentioned this in T235839: Create daily tracking reports for campaign.Nov 25 2019, 4:54 PM
DannyS712 added a subscriber: DannyS712.Jan 15 2020, 6:28 AM
matmarex mentioned this in T24907: Inconsistent values of rev_user_text on pl.wikipedia due to incomplete renames (apparently).Feb 5 2020, 2:36 PM
matmarex added a subscriber: matmarex.
Reedy mentioned this in T244786: rev_comment_id, comment_id join clause.Feb 10 2020, 11:14 PM
gerritbot added a comment.May 8 2020, 3:43 PM

Change 552339 merged by jenkins-bot:
[mediawiki/core@master] Alter revision for actor, comment, and MCR

https://gerrit.wikimedia.org/r/552339

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.32; 2020-05-12).May 8 2020, 4:00 PM
Maintenance_bot removed a project: Patch-For-Review.May 8 2020, 4:11 PM
Jdforrester-WMF updated the task description. (Show Details)May 8 2020, 5:52 PM
• Ladsgroup added a subscriber: Marostegui.May 8 2020, 7:15 PM

I assume next step here is for @Marostegui to apply these two patches in production: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/552339/19/maintenance/archives/patch-archive-MCR.sql and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/552339/19/maintenance/archives/patch-revision-actor-comment-MCR.sql ?

Shall I make a ticket or it already exists?

DannyS712 added a comment.May 8 2020, 7:53 PM
In T215466#6119796, @Ladsgroup wrote:

I assume next step here is for @Marostegui to apply these two patches in production: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/552339/19/maintenance/archives/patch-archive-MCR.sql and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/552339/19/maintenance/archives/patch-revision-actor-comment-MCR.sql ?

Shall I make a ticket or it already exists?

T238966: Apply updates for MCR, actor migration, and content migration, to production wikis.

Jdforrester-WMF added a subtask: T238966: Apply updates for MCR, actor migration, and content migration, to production wikis..May 8 2020, 8:12 PM
daniel mentioned this in T200259: ContribsPager/DeletedContribsPager should use a unique criterion for paging.May 28 2020, 4:51 PM
daniel mentioned this in T238966: Apply updates for MCR, actor migration, and content migration, to production wikis..May 29 2020, 9:37 AM
Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:33 AM
Marostegui changed the status of subtask T238966: Apply updates for MCR, actor migration, and content migration, to production wikis. from Open to Stalled.Jun 15 2020, 8:50 AM
Marostegui changed the status of subtask T238966: Apply updates for MCR, actor migration, and content migration, to production wikis. from Stalled to Open.Jun 16 2020, 7:18 AM
daniel mentioned this in T258480: Query plan changes on enwiki.revision queries with MCR change.Aug 12 2020, 3:26 PM
• AMooney removed Anomie as the assignee of this task.Aug 12 2020, 3:40 PM
Reedy mentioned this in T261059: drop_content_model_info.sql seems to be orphaned.Aug 23 2020, 1:04 AM
Marostegui changed the status of subtask T238966: Apply updates for MCR, actor migration, and content migration, to production wikis. from Open to Stalled.Sep 24 2020, 2:55 PM
daniel added a project: Platform Engineering Roadmap Decision Making.Oct 6 2020, 3:31 PM
nettrom_WMF added a subscriber: nettrom_WMF.Oct 6 2020, 3:32 PM
Naike moved this task from Untriaged to Later on the Platform Engineering Roadmap Decision Making board.Oct 8 2020, 8:42 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:01 PM
Kormat changed the status of subtask T238966: Apply updates for MCR, actor migration, and content migration, to production wikis. from Stalled to Open.Nov 4 2020, 3:44 PM
Prod added a subscriber: Prod.Nov 16 2020, 7:43 PM
Marostegui closed subtask T238966: Apply updates for MCR, actor migration, and content migration, to production wikis. as Resolved.Nov 17 2020, 7:54 AM
Daimona added a subscriber: Daimona.Nov 17 2020, 9:18 AM
RhinosF1 added a subscriber: RhinosF1.Jan 11 2021, 9:50 PM
daniel added a subscriber: daniel.Feb 4 2021, 10:44 AM

Next steps:

  1. Write to rev_actor and rev_comment_id
  2. Backfill rev_actor and rev_comment_id
  3. have a read-new mode, at least for testing
  • We won’t get through with the migration before the 1.36 branch (March?)
    • We should avoid supporting multiple modes in a release
  • But we can start writing to the new schema, and we can also already backfill.
  • The Updater to 1.36 should also backfill.
CCicalese_WMF mentioned this in T275246: Populate rev_actor and rev_comment_id.Feb 19 2021, 9:13 PM
WDoranWMF added a subscriber: WDoranWMF.Mar 1 2021, 7:13 PM

@Marostegui is it correct that step 1.2 of this task is already completed by DBAs? Can we mark step 1 done?

Marostegui added a project: Data-Persistence (work done).Mar 2 2021, 5:43 AM

Correct, this is how the table looks like now:

root@cumin1001:/home/marostegui# mysql.py -hdb1169 enwiki -e "show create table revision\G"
*************************** 1. row ***************************
       Table: revision
Create Table: CREATE TABLE `revision` (
  `rev_id` int(8) unsigned NOT NULL AUTO_INCREMENT,
  `rev_page` int(8) unsigned NOT NULL DEFAULT 0,
  `rev_comment_id` bigint(20) unsigned NOT NULL DEFAULT 0,
  `rev_actor` bigint(20) unsigned NOT NULL DEFAULT 0,
  `rev_timestamp` varbinary(14) NOT NULL DEFAULT '',
  `rev_minor_edit` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `rev_deleted` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `rev_len` int(8) unsigned DEFAULT NULL,
  `rev_parent_id` int(8) unsigned DEFAULT NULL,
  `rev_sha1` varbinary(32) NOT NULL DEFAULT '',
  PRIMARY KEY (`rev_id`),
  KEY `rev_timestamp` (`rev_timestamp`),
  KEY `page_timestamp` (`rev_page`,`rev_timestamp`),
  KEY `rev_page_id` (`rev_page`,`rev_id`),
  KEY `rev_actor_timestamp` (`rev_actor`,`rev_timestamp`,`rev_id`),
  KEY `rev_page_actor_timestamp` (`rev_page`,`rev_actor`,`rev_timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=1009766990 DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED
tstarling updated the task description. (Show Details)Mar 24 2021, 10:55 PM
tstarling updated the task description. (Show Details)
Reedy mentioned this in T252853: updateSearchIndex.php sql error not all tables locked.Mar 27 2021, 3:00 AM
Izno added a subtask: T238615: Adapt mediawiki history for MCR.Apr 27 2021, 5:36 PM
Reedy mentioned this in T284108: Bring labswiki database tables up to date.Jun 1 2021, 9:49 PM
Universal_Omega added a subscriber: Universal_Omega.Jul 4 2021, 7:19 PM
gerritbot added a comment.May 20 2022, 4:27 PM

Change 793845 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] Start clean up of revision_actor_table

https://gerrit.wikimedia.org/r/793845

gerritbot added a project: Patch-For-Review.May 20 2022, 4:27 PM
gerritbot added a comment.May 23 2022, 4:54 PM

Change 793845 merged by jenkins-bot:

[mediawiki/core@master] Start clean up of revision_actor_temp table

https://gerrit.wikimedia.org/r/793845

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.13; 2022-05-23).May 23 2022, 5:00 PM
Maintenance_bot removed a project: Patch-For-Review.May 23 2022, 5:30 PM
gerritbot added a comment.Sep 6 2022, 5:21 PM

Change 830204 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Remove references to revision_actor_temp, aliased as temp_rev_user

https://gerrit.wikimedia.org/r/830204

gerritbot added a project: Patch-For-Review.Sep 6 2022, 5:21 PM
gerritbot added a comment.Sep 7 2022, 8:12 AM

Change 830204 merged by jenkins-bot:

[mediawiki/core@master] Remove references to revision_actor_temp, aliased as temp_rev_user

https://gerrit.wikimedia.org/r/830204

Maintenance_bot removed a project: Patch-For-Review.Sep 7 2022, 8:30 AM
ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.1; 2022-09-12).Sep 7 2022, 9:00 AM
TheresNoTime removed a subscriber: RhinosF1.Dec 15 2022, 11:36 PM
gerritbot added a comment.Dec 30 2022, 6:30 PM

Change 873880 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Stop setting $wgActorTableSchemaMigrationStage

https://gerrit.wikimedia.org/r/873880

gerritbot added a project: Patch-For-Review.Dec 30 2022, 6:30 PM
Zabe added a subscriber: Zabe.Dec 30 2022, 9:19 PM
gerritbot added a comment.Dec 30 2022, 9:19 PM

Change 873882 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/software@master] tables_to_check: drop revision_actor_temp

https://gerrit.wikimedia.org/r/873882

gerritbot added a comment.Jan 2 2023, 3:03 PM

Change 873882 merged by jenkins-bot:

[operations/software@master] tables_to_check: drop revision_actor_temp

https://gerrit.wikimedia.org/r/873882

gerritbot added a comment.Jan 3 2023, 6:10 AM

Change 874466 had a related patch set uploaded (by Func; author: Func):

[mediawiki/core@REL1_39] MigrateActors: Write to revsion table (Follow-up 24115a8)

https://gerrit.wikimedia.org/r/874466

Func added a subscriber: Func.Jan 3 2023, 6:11 AM
gerritbot added a comment.Jan 3 2023, 11:51 AM

Change 874466 merged by jenkins-bot:

[mediawiki/core@REL1_39] MigrateActors: Write to revsion table (Follow-up 24115a8)

https://gerrit.wikimedia.org/r/874466

gerritbot added a comment.Jan 3 2023, 9:03 PM

Change 873880 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop setting $wgActorTableSchemaMigrationStage

https://gerrit.wikimedia.org/r/873880

Stashbot mentioned this in T299954: Write code for handing write and read of rev_comment_id.Jan 3 2023, 9:06 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-03T21:06:40Z] <taavi@deploy1002> Started scap: Backport for [[gerrit:873880|Stop setting $wgActorTableSchemaMigrationStage (T215466)]], [[gerrit:873887|Pin $wgCommentTempTableSchemaMigrationStage to default value (T299954)]], [[gerrit:874418|Pin cu_changes comment migration to old schema (T233004)]]

Stashbot mentioned this in T233004: Update CheckUser for actor and comment table.Jan 3 2023, 9:06 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-03T21:08:30Z] <taavi@deploy1002> taavi and zabe: Backport for [[gerrit:873880|Stop setting $wgActorTableSchemaMigrationStage (T215466)]], [[gerrit:873887|Pin $wgCommentTempTableSchemaMigrationStage to default value (T299954)]], [[gerrit:874418|Pin cu_changes comment migration to old schema (T233004)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet

Stashbot added a comment.Jan 3 2023, 9:15 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-03T21:15:29Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:873880|Stop setting $wgActorTableSchemaMigrationStage (T215466)]], [[gerrit:873887|Pin $wgCommentTempTableSchemaMigrationStage to default value (T299954)]], [[gerrit:874418|Pin cu_changes comment migration to old schema (T233004)]] (duration: 08m 49s)

gerritbot added a comment.Jan 4 2023, 9:45 AM

Change 874878 had a related patch set uploaded (by Func; author: Func):

[mediawiki/core@REL1_39] installer: Split drop action out of the SQL patch for actor migration

https://gerrit.wikimedia.org/r/874878

Milimetric removed a subscriber: Milimetric.Jan 4 2023, 4:39 PM
gerritbot added a comment.Jan 19 2023, 12:21 AM

Change 874878 merged by jenkins-bot:

[mediawiki/core@REL1_39] installer: Split drop action out of the SQL patch for actor migration

https://gerrit.wikimedia.org/r/874878

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2023, 12:30 AM
Zabe updated the task description. (Show Details)Feb 3 2023, 9:47 PM
Aklapper added a project: Data-Engineering-Icebox.Feb 10 2023, 5:44 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL