Page MenuHomePhabricator
Create Task
Maniphest T188327

Deploy refactored actor storage
Closed, ResolvedPublic
Actions

Description

The high-level checklist:

  • 1. Merge the first patch for T167246: Refactor "user" & "user_text" fields into "actor" reference table (adding new schemas and code)
  • 1.1. Check deployed extensions for needed updates.
  • 2. Perform schema change (T188299)
  • Interrupt: Remove read-both (gerrit:461440)
  • 3. Turn the feature flag to "write both, read old". See if stuff breaks.
  • 4. Run the maintenance script(s) to migrate all the old stuff to new stuff.
    • s1
    • s2
    • s3
    • s4
    • s5
    • s6
    • s7
    • s8
    • wikitech
  • 5. Turn the feature flag to "write both, read new". See if stuff breaks.
    • 5.1. Announce the pending change to wikitech-l@ and cloud@, and give time for people to update.
    • 5.2. Make sure all deployed extensions are updated.
    • Beta Cluster
    • Test wikis
    • Group0 wikis
    • Group1 wikis
    • All wikis
  • 6. Turn the feature flag to "new only".
    • Beta Cluster
    • Test wikis
    • Group0 wikis
    • Group1 wikis
    • All wikis
  • 7. Remove old schemas and code
    • 7.1. Update WMCS replicas to no longer reference old schemas (T223406).
    • 7.2. Write and merge patches to remove $wgActorTableSchemaMigrationStage, supporting code, and old schemas (gerrit:525605)
    • 7.3. Submit Schema-change task for WMF production (T233135).

For the cleaning up of revision_actor_temp, see T215466. This and T166733 both block T161671, which in turn is the first step of that task.

Details

SubjectRepoBranchLines +/-
Mostly drop old pre-actor user schemasmediawiki/coremaster+1 K -2 K
Actually create user in AbuseFilterConsequencesTestmediawiki/extensions/AbuseFilterREL1_33+4 -2
Actually create user in AbuseFilterConsequencesTestmediawiki/extensions/AbuseFiltermaster+4 -2
Revert actor changes to populateLogSearch.phpmediawiki/coremaster+43 -64
Update for removal of $wgActorTableSchemaMigrationStagemediawiki/extensions/UserMergemaster+12 -19
Update for removal of $wgActorTableSchemaMigrationStagemediawiki/extensions/ORESmaster+12 -4
Update for removal of $wgActorTableSchemaMigrationStage and $wgCommentTableSchemaMigrationStagemediawiki/extensions/WikimediaMaintenancemaster+6 -24
Set ActorTableSchemaMigrationStage => write-new/read-new on remaining wikisoperations/mediawiki-configmaster+1 -3
Set ActorTableSchemaMigrationStage => write-new/read-new on group 1operations/mediawiki-configmaster+1 -1
Set ActorTableSchemaMigrationStage => write-new/read-new on group 0operations/mediawiki-configmaster+1 -5
Set ActorTableSchemaMigrationStage => write-new/read-new on test wikis & mediawikiwikioperations/mediawiki-configmaster+6 -0
Set actor migration to write-new on Beta Clusteroperations/mediawiki-configmaster+1 -1
Set ActorTableSchemaMigrationStage => write-both/read-new on remaining wikisoperations/mediawiki-configmaster+1 -3
Default $wgActorTableSchemaMigrationStage to SCHEMA_COMPAT_NEWmediawiki/coreREL1_33+266 -26
Default $wgActorTableSchemaMigrationStage to READ_NEWmediawiki/coreREL1_33+30 -11
Default $wgActorTableSchemaMigrationStage to SCHEMA_COMPAT_NEWmediawiki/coremaster+266 -26
Set ActorTableSchemaMigrationStage => write-both/read-new on group 1operations/mediawiki-configmaster+1 -1
Default $wgActorTableSchemaMigrationStage to READ_NEWmediawiki/coremaster+30 -11
Set ActorTableSchemaMigrationStage => write-both/read-new on group 0operations/mediawiki-configmaster+1 -5
Set ActorTableSchemaMigrationStage => write-both/read-new on test wikis & mediawikiwikioperations/mediawiki-configmaster+6 -0
Set actor migration to read-new on Beta Clusteroperations/mediawiki-configmaster+1 -1
Fix slow queries in migrateActors.phpmediawiki/coreREL1_32+74 -34
Fix slow queries in migrateActors.phpmediawiki/coremaster+73 -34
Fix slow queries in migrateActors.phpmediawiki/corewmf/1.33.0-wmf.12+73 -34
Set ActorTableSchemaMigrationStage => write-both/read-old on all wikisoperations/mediawiki-configmaster+1 -3
Set ActorTableSchemaMigrationStage => write-both/read-old on group 1operations/mediawiki-configmaster+1 -1
Set ActorTableSchemaMigrationStage => write-both/read-old on group 0operations/mediawiki-configmaster+1 -3
Set ActorTableSchemaMigrationStage => write-both/read-old on test wikis & mediawikiwikioperations/mediawiki-configmaster+5 -1
Set ActorTableSchemaMigrationStage => OLD on test wikis, mw.orgoperations/mediawiki-configmaster+0 -4
Set ActorTableSchemaMigrationStage => WRITE_BOTH on test wikis, mw.orgoperations/mediawiki-configmaster+8 -0
Set actor and comment schema migrations for Beta Clusteroperations/mediawiki-configmaster+5 -1
Show related patches Customize query in gerrit

Revisions and Commits

rMW MediaWiki
rMW23b5c0891a0d RevDel: Avoid log_search rows with empty values for target_author_actor

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolveddanielT161671 Compacting the revision table
 ResolvedNoneT227047 Complete actor table and comment table migration
 ResolvedAnomieT188327 Deploy refactored actor storage
 ResolvedMarosteguiT188299 Schema change for refactored actor storage
 Resolved• BstormT195747 Create views for the schema change for refactored actor storage
 ResolvedMarosteguiT187089 Fix WMF schemas to not break when comment store goes WRITE_NEW
 ResolvedAnomieT204669 Slow access to Special:Contributions on mediawiki.org (due to enabling actor table WRITE_BOTH mode)
 ResolvedPRODUCTION ERROR• CatropeT204764 DB error in PageTriage: Unknown column 'actor_rev_user.actor_user' in 'on clause'
 ResolvedPRODUCTION ERRORAnomieT204767 Special:ActiveUsers fails with "Error: 1054 Unknown column 'rc_actor' in 'on clause' (10.192.32.110)"
 ResolvedAnomieT209847 Clean up actor renames on mediawikiwiki
 ResolvedPRODUCTION ERRORAnomieT210621 Internal api error: CannotCreateActorException
 ResolvedAnomieT215464 Oversighters can no longer see suppressed contributions past a certain date when using the offender parameter
 ResolvedAnomieT215525 log_search rows with ls_field='target_author_actor' and empty ls_value are created during actor migration
 ResolvedAnomieT223406 Remove reference to fields replaced by the actor table from WMCS views
 ResolvedNiharikaT219324 Update tools maintained by Community-Tech to use new actor storage
 ResolvedMaxSemT219328 Update Event Metrics to use new actor storage
 ResolvedMusikAnimalT223667 Update to use new actor storage
 Resolved• BstormT221339 Missing index on revision_userindex.rev_actor
 ResolvedMilimetricT224134 Finish actor migration in refinery sqoop and refinery source

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.May 1 2019, 5:07 PM

Change 507615 had a related patch set uploaded (by Jforrester; owner: Anomie):
[mediawiki/core@REL1_33] Default $wgActorTableSchemaMigrationStage to SCHEMA_COMPAT_NEW

https://gerrit.wikimedia.org/r/507615

gerritbot added a comment.May 1 2019, 6:02 PM

Change 507614 merged by jenkins-bot:
[mediawiki/core@REL1_33] Default $wgActorTableSchemaMigrationStage to READ_NEW

https://gerrit.wikimedia.org/r/507614

gerritbot added a comment.May 1 2019, 6:02 PM

Change 507615 merged by jenkins-bot:
[mediawiki/core@REL1_33] Default $wgActorTableSchemaMigrationStage to SCHEMA_COMPAT_NEW

https://gerrit.wikimedia.org/r/507615

Jdforrester-WMF mentioned this in T161671: Compacting the revision table.May 1 2019, 6:47 PM
ReleaseTaggerBot edited projects, added MW-1.33-notes; removed MW-1.33-notes (1.33.0-wmf.25; 2019-04-09).May 1 2019, 7:00 PM
gerritbot added a comment.May 13 2019, 1:33 PM

Change 509844 had a related patch set uploaded (by Anomie; owner: Anomie):
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-both/read-new on remaining wikis

https://gerrit.wikimedia.org/r/509844

gerritbot added a comment.May 13 2019, 1:34 PM

Change 509844 merged by jenkins-bot:
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-both/read-new on remaining wikis

https://gerrit.wikimedia.org/r/509844

Stashbot added a comment.May 13 2019, 1:36 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-13T13:36:37Z] <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-new on all wikis (T188327) (duration: 00m 50s)

gerritbot added a comment.May 13 2019, 4:04 PM

Change 509883 had a related patch set uploaded (by Anomie; owner: Anomie):
[operations/mediawiki-config@master] Set actor migration to write-new on Beta Cluster

https://gerrit.wikimedia.org/r/509883

gerritbot added a comment.May 13 2019, 4:05 PM

Change 509883 merged by jenkins-bot:
[operations/mediawiki-config@master] Set actor migration to write-new on Beta Cluster

https://gerrit.wikimedia.org/r/509883

Jdforrester-WMF removed a project: Patch-For-Review.May 13 2019, 5:19 PM
Jdforrester-WMF updated the task description. (Show Details)
Anomie mentioned this in T223151: Review special replica partitioning of certain tables by `xx_user`.May 13 2019, 5:50 PM
Anomie mentioned this in T222224: RFC: Normalize MediaWiki link tables.May 14 2019, 8:58 PM
gerritbot added a comment.May 15 2019, 1:18 PM

Change 510503 had a related patch set uploaded (by Anomie; owner: Anomie):
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-new/read-new on test wikis & mediawikiwiki

https://gerrit.wikimedia.org/r/510503

gerritbot added a project: Patch-For-Review.May 15 2019, 1:18 PM

Change 510503 merged by jenkins-bot:
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-new/read-new on test wikis & mediawikiwiki

https://gerrit.wikimedia.org/r/510503

Stashbot added a comment.May 15 2019, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-15T13:21:15Z] <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-new/read-new on testwikis and mediawikiwiki (T188327) (duration: 00m 57s)

Anomie removed a project: Patch-For-Review.May 15 2019, 1:21 PM
Anomie updated the task description. (Show Details)
Anomie mentioned this in T223406: Remove reference to fields replaced by the actor table from WMCS views.May 15 2019, 6:51 PM
Anomie updated the task description. (Show Details)May 15 2019, 6:55 PM
gerritbot added a comment.May 20 2019, 3:10 PM

Change 511444 had a related patch set uploaded (by Anomie; owner: Anomie):
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-new/read-new on group 0

https://gerrit.wikimedia.org/r/511444

gerritbot added a project: Patch-For-Review.May 20 2019, 3:10 PM

Change 511444 merged by jenkins-bot:
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-new/read-new on group 0

https://gerrit.wikimedia.org/r/511444

Stashbot added a comment.May 20 2019, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-20T15:13:08Z] <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-new/read-new on group 0 (T188327) (duration: 00m 55s)

Anomie removed a project: Patch-For-Review.May 20 2019, 3:25 PM
Anomie updated the task description. (Show Details)
gerritbot added a comment.May 29 2019, 3:02 PM

Change 513139 had a related patch set uploaded (by Anomie; owner: Anomie):
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-new/read-new on group 1

https://gerrit.wikimedia.org/r/513139

gerritbot added a project: Patch-For-Review.May 29 2019, 3:02 PM
gerritbot added a comment.May 29 2019, 3:04 PM

Change 513139 merged by jenkins-bot:
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-new/read-new on group 1

https://gerrit.wikimedia.org/r/513139

Stashbot added a comment.May 29 2019, 3:06 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-29T15:06:12Z] <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-new/read-new on group 1 (T188327) (duration: 00m 57s)

Maintenance_bot removed a project: Patch-For-Review.May 29 2019, 3:10 PM
jcrespo awarded a token.May 29 2019, 3:11 PM
Anomie updated the task description. (Show Details)May 29 2019, 3:23 PM
Daimona subscribed.May 31 2019, 12:11 PM
gerritbot added a comment.Jun 3 2019, 12:53 PM

Change 513996 had a related patch set uploaded (by Anomie; owner: Anomie):
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-new/read-new on remaining wikis

https://gerrit.wikimedia.org/r/513996

gerritbot added a project: Patch-For-Review.Jun 3 2019, 12:53 PM

Change 513996 merged by jenkins-bot:
[operations/mediawiki-config@master] Set ActorTableSchemaMigrationStage => write-new/read-new on remaining wikis

https://gerrit.wikimedia.org/r/513996

Anomie removed a project: Patch-For-Review.Jun 3 2019, 12:55 PM
Anomie updated the task description. (Show Details)
Stashbot added a comment.Jun 3 2019, 12:56 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-03T12:55:58Z] <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-new/read-new on remaining wikis (T188327) (duration: 00m 48s)

Daimona mentioned this in T224930: No contributions shown for any user in Global user contribution tool.Jun 3 2019, 7:31 PM
kzimmerman subscribed.Jun 4 2019, 5:15 PM
bd808 added a subtask: T225007: Update operator/user manual description of database tables changed for refactored actor storage.Jun 4 2019, 5:34 PM
• Bstorm closed subtask T223406: Remove reference to fields replaced by the actor table from WMCS views as Resolved.Jun 4 2019, 6:31 PM
Anomie updated the task description. (Show Details)Jun 4 2019, 6:56 PM
jcrespo mentioned this in T16862: Allow global renaming of global users.Jun 5 2019, 10:26 AM
DerHexer subscribed.Jun 5 2019, 2:40 PM
CCicalese_WMF added a parent task: T227047: Complete actor table and comment table migration.Jul 1 2019, 11:48 PM
• WDoranWMF moved this task from Doing to Team 1 on the Platform Team Workboards board.Jul 5 2019, 6:35 PM
• WDoranWMF edited projects, added Platform Team Workboards (Team 1); removed Platform Team Workboards (Doing).
• WDoranWMF moved this task from Backlog to Doing on the Platform Team Workboards (Team 1) board.
• WDoranWMF edited projects, added Platform Engineering (Revision Storage Schema Improvements); removed Platform Engineering (Needs Cleaning - Code Health (TEC13)).Jul 9 2019, 7:07 PM
• WDoranWMF edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Team Workboards (Team 1).Jul 17 2019, 12:14 AM
Anomie moved this task from Inbox to Ready (WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.Jul 17 2019, 8:03 PM
• WDoranWMF moved this task from Ready (WIP:5) to Later on the Platform Team Workboards (Clinic Duty Team) board.Jul 17 2019, 8:10 PM
• WDoranWMF moved this task from Later to Ready (WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.Jul 22 2019, 11:13 PM
• WDoranWMF triaged this task as High priority.Jul 23 2019, 3:54 PM
Anomie moved this task from Ready (WIP:5) to Doing(WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.Jul 24 2019, 5:20 PM
gerritbot added a comment.Jul 25 2019, 5:38 PM

Change 525602 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/extensions/ORES@master] Update for removal of $wgActorTableSchemaMigrationStage

https://gerrit.wikimedia.org/r/525602

gerritbot added a project: Patch-For-Review.Jul 25 2019, 5:38 PM

Change 525603 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/extensions/UserMerge@master] Update for removal of $wgActorTableSchemaMigrationStage

https://gerrit.wikimedia.org/r/525603

gerritbot added a comment.Jul 25 2019, 5:38 PM

Change 525604 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/extensions/WikimediaMaintenance@master] Update for removal of $wgActorTableSchemaMigrationStage and $wgCommentTableSchemaMigrationStage

https://gerrit.wikimedia.org/r/525604

gerritbot added a comment.Jul 25 2019, 5:38 PM

Change 525605 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@master] Mostly drop old pre-actor user schemas

https://gerrit.wikimedia.org/r/525605

Anomie updated the task description. (Show Details)Jul 25 2019, 5:57 PM
Anomie moved this task from Doing(WIP:5) to Waiting for Review on the Platform Team Workboards (Clinic Duty Team) board.
• WDoranWMF moved this task from Revision Storage Schema Improvements to mop on the Platform Engineering board.Jul 26 2019, 7:11 PM
• WDoranWMF edited projects, added Core Platform Team Initiatives (Revision Storage Schema Improvements); removed Platform Engineering (Revision Storage Schema Improvements).
gerritbot added a comment.Jul 29 2019, 4:11 AM

Change 525602 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Update for removal of $wgActorTableSchemaMigrationStage

https://gerrit.wikimedia.org/r/525602

gerritbot added a comment.Jul 29 2019, 4:11 AM

Change 525603 merged by jenkins-bot:
[mediawiki/extensions/UserMerge@master] Update for removal of $wgActorTableSchemaMigrationStage

https://gerrit.wikimedia.org/r/525603

gerritbot added a comment.Jul 29 2019, 4:11 AM

Change 525604 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMaintenance@master] Update for removal of $wgActorTableSchemaMigrationStage and $wgCommentTableSchemaMigrationStage

https://gerrit.wikimedia.org/r/525604

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.16; 2019-07-30); removed MW-1.34-notes (1.34.0-wmf.4; 2019-05-07).Jul 29 2019, 5:00 AM
gerritbot added a comment.Aug 7 2019, 9:44 PM

Change 528952 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@master] Revert actor changes to populateLogSearch.php

https://gerrit.wikimedia.org/r/528952

Anomie mentioned this in T2323: Edits from various users active before 2003 are missing from Special:Contributions.Aug 20 2019, 4:05 PM
matej_suchanek mentioned this in T31757: Incomplete user renames (and other username discrepancies) (tracking).Aug 21 2019, 1:54 PM
gerritbot added a comment.Aug 28 2019, 4:29 AM

Change 528952 merged by jenkins-bot:
[mediawiki/core@master] Revert actor changes to populateLogSearch.php

https://gerrit.wikimedia.org/r/528952

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.21; 2019-09-03); removed MW-1.34-notes (1.34.0-wmf.16; 2019-07-30).Aug 28 2019, 5:00 AM
hashar mentioned this in T231965: AbuseFilter tests fail on REL1_33: CannotCreateActorException: Cannot create an actor for a usable name that is not an existing user.Sep 4 2019, 6:59 AM
gerritbot added a comment.Sep 4 2019, 7:36 AM

Change 534293 had a related patch set uploaded (by Hashar; owner: Anomie):
[mediawiki/extensions/AbuseFilter@REL1_33] Actually create user in AbuseFilterConsequencesTest

https://gerrit.wikimedia.org/r/534293

gerritbot added a comment.Sep 4 2019, 12:27 PM

Change 534293 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@REL1_33] Actually create user in AbuseFilterConsequencesTest

https://gerrit.wikimedia.org/r/534293

gerritbot added a comment.Sep 9 2019, 5:09 PM

Change 525605 merged by jenkins-bot:
[mediawiki/core@master] Mostly drop old pre-actor user schemas

https://gerrit.wikimedia.org/r/525605

Anomie removed a project: Patch-For-Review.Sep 9 2019, 5:19 PM
Anomie updated the task description. (Show Details)
Anomie moved this task from Waiting for Review to Waiting for deployment on the Platform Team Workboards (Clinic Duty Team) board.Sep 11 2019, 9:14 PM
Anomie closed this task as Resolved.Sep 17 2019, 5:40 PM
Anomie updated the task description. (Show Details)
Anomie moved this task from Waiting for deployment to Done on the Platform Team Workboards (Clinic Duty Team) board.Sep 17 2019, 5:42 PM

Note the final Wikimedia production schema changes are tracked in T233135: Schema change for refactored actor and comment storage, and cleanup of revision_actor_temp in T215466: Remove revision_comment_temp and revision_actor_temp. Changes for any extensions should be tracked in their own tasks.

Iflorez mentioned this in T234560: identify a reliable user field for use with the revision, page...maybe also the actor table.Oct 3 2019, 7:23 PM
Bugreporter mentioned this in T238997: dewiki.revision has no registered users' edits since 20191101000000.Nov 23 2019, 10:23 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:40 PM
Aklapper mentioned this in T267543: Upgrade from 1.21 to 1.35 fails: patch-drop-image-user-fields.sql tries to drop img_user_timestamp which may not exist on some upgrades.Nov 9 2020, 11:53 AM
Aklapper mentioned this in T24907: Inconsistent values of rev_user_text on pl.wikipedia due to incomplete renames (apparently).Mar 1 2021, 11:42 AM
Aklapper mentioned this in T28942: edits by a registered user where the rev_user field in the revision table or the ar_user field in the archive table are 0 do not move when that user is renamed.
Izno removed a subtask: T225007: Update operator/user manual description of database tables changed for refactored actor storage.Apr 27 2021, 4:44 PM
Izno removed a subtask: T167246: Refactor "user" & "user_text" fields into "actor" reference table.Apr 27 2021, 4:51 PM
Seb35 mentioned this in T217972: log_user_text is missing for some logs in fawiki.Feb 22 2022, 11:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits