Page MenuHomePhabricator

Populate rev_actor and rev_comment_id
Open, Needs TriagePublic1 Estimated Story Points

Description

This includes steps 2-4 of T215466:

  • 2. Introduce a migration from using revision_comment_temp and revision_actor_temp (old) to rev_comment_id and rev_actor (new).
    • 2.1. Write and merge the MediaWiki patch, assuming it wasn't included in 1.1.
    • 2.2. Check for extensions assuming the temp tables exist.
  • 3. Set migration stage to write-both/read-old.
  • 4. Run the maintenance script at WMF to backfill rev_comment_id and rev_actor.

And finally:

  • Add the migration script to the MW installer/updater.

Keeping in mind:

We should avoid supporting multiple modes in a release.

And:

[…] on the principle that we should fully test our code in production before forcing non-WMF users to use it.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+50 -50
operations/puppetproduction+25 -25
operations/mediawiki-configmaster+1 -3
operations/mediawiki-configmaster+1 -0
operations/mediawiki-configmaster+1 -0
operations/mediawiki-configmaster+1 -1
operations/puppetproduction+25 -25
operations/puppetproduction+7 -23
operations/mediawiki-configmaster+1 -4
operations/mediawiki-configmaster+1 -0
operations/mediawiki-configmaster+1 -0
operations/mediawiki-configmaster+1 -0
operations/mediawiki-configmaster+1 -1
mediawiki/extensions/FlaggedRevsmaster+1 -1
operations/mediawiki-configmaster+1 -5
operations/mediawiki-configmaster+4 -0
mediawiki/coremaster+5 -0
operations/mediawiki-configmaster+8 -0
mediawiki/corewmf/1.38.0-wmf.9+15 -0
mediawiki/coremaster+15 -0
mediawiki/coremaster+893 -558
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 684142 merged by jenkins-bot:

[mediawiki/core@master] Schema migration for revison_actor_temp table removal

https://gerrit.wikimedia.org/r/684142

Hey there. This task is proposed as a blocker to MediaWiki 1.37, which will be cut in less than three weeks' time. Please consider whether this will make that deadline, and if not, move it to block the MediaWiki 1.38 release (MW-1.38-release) or remove as a blocker entirely.

Hi, what's the status of this? I would like to finish the migration and if there is anything I can help, let me know.

I updated the checklist. The next step is to do the migration in production. Wikidata reads revision rows from foreign databases, which implies that the configuration for all wikis should be kept in sync. So:

  • Set $wgActorTableSchemaMigrationStage = SCHEMA_COMPAT_WRITE_TEMP_AND_NEW | SCHEMA_COMPAT_READ_TEMP
  • Run maintenance/migrateRevisionActorTemp.php on all wikis
  • Set $wgActorTableSchemaMigrationStage = SCHEMA_COMPAT_WRITE_TEMP_AND_NEW | SCHEMA_COMPAT_READ_NEW

I think that will resolve this task. The task description says that the migration should also be added to the updater, but I think that can wait until after production has been migrated, i.e. step 7 of T215466, on the principle that we should fully test our code in production before forcing non-WMF users to use it.

Thanks. I will do the switching the config on Monday and start the backfill script in test wikis (then small wikis and so on). I hope that's fine with you.

Change 739633 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Revert \"Stop setting wgActorTableSchemaMigrationStage, no longer read in core\"

https://gerrit.wikimedia.org/r/739633

Change 739633 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert \"Stop setting wgActorTableSchemaMigrationStage, no longer read in core\"

https://gerrit.wikimedia.org/r/739633

Mentioned in SAL (#wikimedia-operations) [2021-11-18T01:35:29Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: NOOP - Config: [[gerrit:739633|Revert "Stop setting wgActorTableSchemaMigrationStage, no longer read in core" (T275246)]] (duration: 01m 04s)

Mentioned in SAL (#wikimedia-releng) [2021-11-18T01:42:21Z] <Amir1> ladsgroup@deployment-deploy01:~$ foreachwikiindblist all-labs maintenance/migraeRevisionActorTemp.php (T275246)

Change 739695 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] maintenance: Add waitForReplication and sleep in migrateRevisionActorTemp

https://gerrit.wikimedia.org/r/739695

Change 739695 merged by jenkins-bot:

[mediawiki/core@master] maintenance: Add waitForReplication and sleep in migrateRevisionActorTemp

https://gerrit.wikimedia.org/r/739695

Change 739636 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.38.0-wmf.9] maintenance: Add waitForReplication and sleep in migrateRevisionActorTemp

https://gerrit.wikimedia.org/r/739636

Change 739636 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.9] maintenance: Add waitForReplication and sleep in migrateRevisionActorTemp

https://gerrit.wikimedia.org/r/739636

Mentioned in SAL (#wikimedia-operations) [2021-11-18T07:35:47Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.9/maintenance/migrateRevisionActorTemp.php: Backport: [[gerrit:739636|maintenance: Add waitForReplication and sleep in migrateRevisionActorTemp (T275246)]] (duration: 01m 04s)

Change 739935 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] maintenance: Add ability to start from rev_id to migrateRevisionActorTemp

https://gerrit.wikimedia.org/r/739935

Change 739935 merged by jenkins-bot:

[mediawiki/core@master] maintenance: Add ability to start from rev_id to migrateRevisionActorTemp

https://gerrit.wikimedia.org/r/739935

Change 740807 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Set test wikis to write both for actor temp table migration

https://gerrit.wikimedia.org/r/740807

Change 740807 merged by jenkins-bot:

[operations/mediawiki-config@master] Set test wikis to write both for actor temp table migration

https://gerrit.wikimedia.org/r/740807

Mentioned in SAL (#wikimedia-operations) [2021-11-23T11:02:21Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:740807|Set test wikis to write both for actor temp table migration (T275246)]] (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2021-11-23T11:08:14Z] <Amir1> start of mwscript migrateRevisionActorTemp.php --wiki=testwiki --sleep=5 (T275246)

Change 741082 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Set actor migration to write both on all wikis

https://gerrit.wikimedia.org/r/741082

Change 741082 merged by jenkins-bot:

[operations/mediawiki-config@master] Set actor migration to write both on all wikis

https://gerrit.wikimedia.org/r/741082

Mentioned in SAL (#wikimedia-operations) [2021-11-24T08:51:31Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:741082|Set actor migration to write both on all wikis (T275246)]] (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2021-11-24T16:13:30Z] <Amir1> start of "foreachwikiindblist s3 migrateRevisionActorTemp.php --sleep=2" in mwmaint1002 in a screen. It will take a month or so (T275246)

Change 701218 merged by jenkins-bot:

[mediawiki/extensions/FlaggedRevs@master] Fix the name of the new contributions index

https://gerrit.wikimedia.org/r/701218

Mentioned in SAL (#wikimedia-operations) [2022-01-03T04:21:01Z] <Amir1> start of running populating actor in revision table on rest of sections. It will take two months to finish (T275246)

Mentioned in SAL (#wikimedia-operations) [2021-11-24T16:13:30Z] <Amir1> start of "foreachwikiindblist s3 migrateRevisionActorTemp.php --sleep=2" in mwmaint1002 in a screen. It will take a month or so (T275246)

s3 is fully done now. Took two months and 6 days.

Krinkle added a subscriber: Krinkle.

Last year, [change 684142[(https://gerrit.wikimedia.org/r/c/mediawiki/core/+/684142) implemented the migration script. Tim suggested at T275246#7512017 that we add it tothe installer/updater after we've succesfully completed the migration on WMF's databases, which is (also) partly tracked at T215466, which has been dormant for the past twelve months.

I'm untagging this from the release blockers for now as it isn't something we can finish or backport through code alone. When the migration is done, the task already has an open action item to update the installer at that time, which doesn't need to be scheduled or blocking any particular release.

I've updated the task description with the keypoints from Daniel and Tim's comments. The release of this was originally scheduled for 1.36. I've removed it as blocker for now since we want to complete this at WMF first. Moving to 1.39 as such since this isn't something we can (nor should) rush and backport through code changes alone.

Note that this is still ongoing, not blocked or stalled. I have been checking the script every couple of days to make sure it's not stopped. It's just looong and doing a lot of work. My estimation is that it will be finished in May or June. Then we can switch the read and start dropping the temp table for actor.

Keep it in mind that we haven't done anything for comment yet.

s2 is done, only s1, s4 (mostly done) and s8 left.

I made s1 and s8 slightly faster.

Change 779166 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] labs: Set actor migration to read new

https://gerrit.wikimedia.org/r/779166

Change 779166 merged by jenkins-bot:

[operations/mediawiki-config@master] labs: Set actor migration to read new

https://gerrit.wikimedia.org/r/779166

Only wikidata left and it's mostly done. I'll slowly start read new in production once wikidata is done.

Change 783845 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] maintain-views: Drop views on revision_actor_temp

https://gerrit.wikimedia.org/r/783845

Change 784685 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] ActorMigration: Start reading from rev_actor field in group0

https://gerrit.wikimedia.org/r/784685

s8 is also done. Nothing left. I'm going to enable read new next week.

Change 784685 merged by jenkins-bot:

[operations/mediawiki-config@master] ActorMigration: Start reading from rev_actor field in group0

https://gerrit.wikimedia.org/r/784685

Mentioned in SAL (#wikimedia-operations) [2022-04-25T07:11:14Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:784685|ActorMigration: Start reading from rev_actor field in group0 (T275246)]] (duration: 00m 50s)

Ran a double check on all dbs and confirm nothing left to do in production.

Change 785948 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] ActorMigration: Read from rev_actor field in all of small wikis

https://gerrit.wikimedia.org/r/785948

Change 785948 merged by jenkins-bot:

[operations/mediawiki-config@master] ActorMigration: Read from rev_actor field in all of small wikis

https://gerrit.wikimedia.org/r/785948

Mentioned in SAL (#wikimedia-operations) [2022-04-25T23:01:48Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:785948|ActorMigration: Read from rev_actor field in all of small wikis (T275246)]] (duration: 00m 57s)

Change 786341 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Set actor migration to read new for medium wikis

https://gerrit.wikimedia.org/r/786341

Change 786341 merged by jenkins-bot:

[operations/mediawiki-config@master] Set actor migration to read new for medium wikis

https://gerrit.wikimedia.org/r/786341

Mentioned in SAL (#wikimedia-operations) [2022-04-26T16:32:09Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786341|Set actor migration to read new for medium wikis (T275246)]] (duration: 02m 01s)

Change 786936 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Set actor migration to READ NEW everywhere

https://gerrit.wikimedia.org/r/786936

Change 786936 merged by jenkins-bot:

[operations/mediawiki-config@master] Set actor migration to READ NEW everywhere

https://gerrit.wikimedia.org/r/786936

Mentioned in SAL (#wikimedia-operations) [2022-04-27T12:07:10Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:786936|Set actor migration to READ NEW everywhere (T275246)]] (duration: 00m 53s)

This is now reading from rev_actor field everywhere. Next step is to remove the view and then we can start to stop writing to the temp table.

Change 783845 merged by Ladsgroup:

[operations/puppet@production] maintain-views: Drop views on revision_actor_temp

https://gerrit.wikimedia.org/r/783845

Change 789271 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] dbproxy: depool clouddb1013-1017 to update views

https://gerrit.wikimedia.org/r/789271

Change 789271 merged by Razzi:

[operations/puppet@production] dbproxy: depool clouddb1013-1017 to update views

https://gerrit.wikimedia.org/r/789271

Change 789291 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] dbproxy: Repool the old batch, Depool the new one

https://gerrit.wikimedia.org/r/789291

Change 789291 merged by Ladsgroup:

[operations/puppet@production] dbproxy: Repool the old batch, Depool the new one

https://gerrit.wikimedia.org/r/789291

Change 789308 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] dbproxy: Repool clouddb10(17|18|19|20)

https://gerrit.wikimedia.org/r/789308

Change 789308 merged by Ladsgroup:

[operations/puppet@production] dbproxy: Repool clouddb10(17|18|19|20)

https://gerrit.wikimedia.org/r/789308

We are ready to stop writing to the temp table now

Change 789314 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] labs: Set actor migration to write new only

https://gerrit.wikimedia.org/r/789314

Change 789314 merged by jenkins-bot:

[operations/mediawiki-config@master] labs: Set actor migration to write new only

https://gerrit.wikimedia.org/r/789314

confirmed it's working in beta as expected. Moving to group0!!!!!11%$^%&

Change 789558 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Stop writing to temp actor table in group0

https://gerrit.wikimedia.org/r/789558

Change 789558 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop writing to temp actor table in group0

https://gerrit.wikimedia.org/r/789558

Mentioned in SAL (#wikimedia-operations) [2022-05-05T18:34:37Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:789558|Stop writing to temp actor table in group0 (T275246)]] (duration: 00m 50s)

Working as expected:

wikiadmin@10.64.0.44(mediawikiwiki)> select * from revision order by rev_id desc limit 5;
+---------+----------+----------------+-----------+----------------+----------------+-------------+---------+---------------+---------------------------------+
| rev_id  | rev_page | rev_comment_id | rev_actor | rev_timestamp  | rev_minor_edit | rev_deleted | rev_len | rev_parent_id | rev_sha1                        |
+---------+----------+----------------+-----------+----------------+----------------+-------------+---------+---------------+---------------------------------+
| 5198358 |     6905 |              0 |  18548471 | 20220505183752 |              0 |           0 |    4441 |       4693495 | rpok3xa7dpfgexkz7hh572c4uyrfpb0 |
| 5198357 |   127100 |              0 |  18548471 | 20220505183751 |              0 |           0 |      50 |       4549557 | 08ycj4qw81r2hk3zm95atcfigxibz0w |
| 5198356 |  1528085 |              0 |  18365305 | 20220505183359 |              0 |           0 |     450 |       5148536 | n9ijs4rmktse5fwujot3hb0lgmkre7n |
| 5198355 |    82610 |              0 |  18365305 | 20220505183206 |              0 |           0 |    8350 |       5198353 | 7m2uk076lkt8g6b87y0rnloxs0t8ofj |
| 5198354 |  1541652 |              0 |  18365305 | 20220505183205 |              0 |           0 |     276 |             0 | aildubs9ltc6v3ct0a3b72hpuoulnwx |
+---------+----------+----------------+-----------+----------------+----------------+-------------+---------+---------------+---------------------------------+
5 rows in set (0.000 sec)

wikiadmin@10.64.0.44(mediawikiwiki)> select * from revision_actor_temp order by revactor_rev desc limit 5;
+--------------+----------------+--------------------+---------------+
| revactor_rev | revactor_actor | revactor_timestamp | revactor_page |
+--------------+----------------+--------------------+---------------+
|      5198356 |       18365305 | 20220505183359     |       1528085 |
|      5198355 |       18365305 | 20220505183206     |         82610 |
|      5198354 |       18365305 | 20220505183205     |       1541652 |
|      5198353 |       18365305 | 20220505183137     |         82610 |
|      5198352 |       18365305 | 20220505183135     |       1541651 |
+--------------+----------------+--------------------+---------------+
5 rows in set (0.001 sec)

Will push it to every wiki next week if no errors show up.

Change 790020 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Stop writing to rev_actor_temp table in group1

https://gerrit.wikimedia.org/r/790020

Nothing has written to the table since we disabled it:

root@db1157:/srv/sqldata/mediawikiwiki# ls -l | grep -i temp
-rw-rw---- 1 mysql mysql       2524 Jun  7  2021 revision_actor_temp.frm
-rw-rw---- 1 mysql mysql  922746880 May  5 18:39 revision_actor_temp.ibd

Change 790020 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop writing to rev_actor_temp table in group1

https://gerrit.wikimedia.org/r/790020

Mentioned in SAL (#wikimedia-operations) [2022-05-09T04:40:02Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:790020|Stop writing to rev_actor_temp table in group1 (T275246)]] (duration: 05m 06s)

Change 790618 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Stop writing to revision_actor_temp table everywhere

https://gerrit.wikimedia.org/r/790618

Change 790618 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop writing to revision_actor_temp table everywhere

https://gerrit.wikimedia.org/r/790618

Mentioned in SAL (#wikimedia-operations) [2022-05-10T08:05:02Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:790618|Stop writing to revision_actor_temp table everywhere (T275246)]] (duration: 00m 48s)