Page MenuHomePhabricator

Update CheckUser for actor and comment table
Open, MediumPublic

Description

Current CheckUser extension does not use actor table and comment table.

Related tasks:

Details

ProjectBranchLines +/-Subject
mediawiki/extensions/CheckUsermaster+12 -0
operations/mediawiki-configmaster+1 -90
operations/mediawiki-configmaster+82 -0
operations/mediawiki-configmaster+0 -82
mediawiki/extensions/CheckUsermaster+66 -0
mediawiki/extensions/CheckUsermaster+17 -8
mediawiki/extensions/CheckUserwmf/1.39.0-wmf.12+2 -2
operations/mediawiki-configmaster+90 -6
mediawiki/extensions/CheckUserwmf/1.39.0-wmf.13+2 -2
mediawiki/extensions/CheckUsermaster+2 -2
operations/mediawiki-configmaster+6 -90
operations/mediawiki-configmaster+90 -6
operations/mediawiki-configmaster+3 -0
operations/mediawiki-configmaster+2 -0
mediawiki/extensions/CheckUsermaster+5 -1
operations/mediawiki-configmaster+2 -0
mediawiki/extensions/CheckUsermaster+147 -1
mediawiki/extensions/CheckUsermaster+1 -1
operations/mediawiki-configmaster+5 -0
mediawiki/extensions/CheckUsermaster+7 -3
mediawiki/extensions/CheckUsermaster+97 -19
mediawiki/extensions/CheckUsermaster+208 -38
mediawiki/extensions/CheckUsermaster+31 -2
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Hi @Ladsgroup :) Thanks for caring !
You'll find some minimal doc here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Edit_data_loading with a link to the code.
Please ping in IRC or slack if you wish more details.

Zabe renamed this task from Schema changes for `cu_changes` and `cu_log` table to Normalize CheckUser tables.Mar 11 2022, 3:15 AM
Zabe changed the task status from Stalled to Open.
Zabe updated the task description. (Show Details)
Zabe removed a subscriber: AMooney.

Changed the task from being a weird mix of requesing an application of a schema change on production and a task where the updating is coordinated. Requesting the application of a schema to production can be done in a subtask when we are at that point.

I also dropped the cuc_logid proposal, because it is quite unrelated. It is already covered by T253796.

I am going to work on a patch doing this. The other patch is more than two years old and somewhat outdated.

Zabe renamed this task from Normalize CheckUser tables to Update CheckUser for actor and comment table.Mar 11 2022, 3:29 AM

Change 769824 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] Add actor and comment columns to CheckUser tables

https://gerrit.wikimedia.org/r/769824

Change 769824 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Add actor and comment columns to cu_changes

https://gerrit.wikimedia.org/r/769824

As I understand, the patch just adds the columns but is there now a plan to migrate the data? We (data engineering) would like to be kept in the loop as closely as possible here.

As I understand, the patch just adds the columns but is there now a plan to migrate the data? We (data engineering) would like to be kept in the loop as closely as possible here.

There is a plan, but no code yet. Any actual migration is currently stalled on T303603.

Change 773650 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] WIP: Add actor migration code

https://gerrit.wikimedia.org/r/773650

Change 775336 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] Add actor migration code for SCHEMA_COMPAT_WRITE_BOTH

https://gerrit.wikimedia.org/r/775336

Change 537256 abandoned by Zabe:

[mediawiki/extensions/CheckUser@master] Schema change: Add columns cuc_log_id and ActorMigrate, CommentStore related

Reason:

https://gerrit.wikimedia.org/r/537256

Change 775336 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Add actor migration code for SCHEMA_COMPAT_WRITE_BOTH

https://gerrit.wikimedia.org/r/775336

Change 776224 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] Use ActorStore to fetch actor id

https://gerrit.wikimedia.org/r/776224

Change 776224 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Use ActorStore to fetch actor id

https://gerrit.wikimedia.org/r/776224

Change 776249 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] Add population script for cuc_actor

https://gerrit.wikimedia.org/r/776249

Change 776250 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Pin CheckUser actor migration to old schema

https://gerrit.wikimedia.org/r/776250

Change 776252 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] Change actor migration to write both

https://gerrit.wikimedia.org/r/776252

Change 776250 merged by jenkins-bot:

[operations/mediawiki-config@master] Pin CheckUser actor migration to old schema

https://gerrit.wikimedia.org/r/776250

Mentioned in SAL (#wikimedia-operations) [2022-04-05T13:17:02Z] <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:776250|Pin CheckUser actor migration to old schema (T233004)]] (duration: 00m 54s)

Change 776252 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Change actor migration to write both

https://gerrit.wikimedia.org/r/776252

Change 776249 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Add population script for cuc_actor

https://gerrit.wikimedia.org/r/776249

Change 779078 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Start writing to cuc_actor in guwwiki and shnwikivoyage

https://gerrit.wikimedia.org/r/779078

Change 779078 merged by jenkins-bot:

[operations/mediawiki-config@master] Start writing to cuc_actor in guwwiki and shnwikivoyage

https://gerrit.wikimedia.org/r/779078

Mentioned in SAL (#wikimedia-operations) [2022-04-11T20:24:40Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 17c8c17: Start writing to cuc_actor in guwwiki and shnwikivoyage (T233004) (duration: 00m 51s)

Change 779912 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] Acquire fresh actor id for anons

https://gerrit.wikimedia.org/r/779912

Change 779922 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Revert "Revert "Start writing to cuc_actor in guwwiki and shnwikivoyage""

https://gerrit.wikimedia.org/r/779922

Change 779912 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Acquire fresh actor id for anons

https://gerrit.wikimedia.org/r/779912

Change 782195 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] SpecialCheckUser: add support for actor migration read new

https://gerrit.wikimedia.org/r/782195

Change 779922 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Revert "Start writing to cuc_actor in guwwiki and shnwikivoyage""

https://gerrit.wikimedia.org/r/779922

Mentioned in SAL (#wikimedia-operations) [2022-05-02T20:30:36Z] <catrope@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:779922|Revert "Revert "Start writing to cuc_actor in guwwiki and shnwikivoyage"" (T233004)]] (duration: 00m 47s)

Change 797294 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Start writing to cuc_actor in s3 and labtestwiki

https://gerrit.wikimedia.org/r/797294

Change 797312 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Start writing to cuc_actor in test wikis

https://gerrit.wikimedia.org/r/797312

Change 797312 merged by jenkins-bot:

[operations/mediawiki-config@master] Start writing to cuc_actor in test wikis

https://gerrit.wikimedia.org/r/797312

Mentioned in SAL (#wikimedia-operations) [2022-05-23T20:21:17Z] <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:797312|Start writing to cuc_actor in test wikis (T233004)]] (duration: 00m 50s)

Change 797294 merged by jenkins-bot:

[operations/mediawiki-config@master] Start writing to cuc_actor in s3, kcgwiki and labtestwiki

https://gerrit.wikimedia.org/r/797294

Mentioned in SAL (#wikimedia-operations) [2022-05-24T21:06:18Z] <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:797294|Start writing to cuc_actor in s3, kcgwiki and labtestwiki (T233004)]] (duration: 00m 52s)

Change 798993 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] Acquire fresh actor id

https://gerrit.wikimedia.org/r/798993

Change 798813 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Revert "Start writing to cuc_actor in s3, kcgwiki and labtestwiki"

https://gerrit.wikimedia.org/r/798813

Change 798813 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Start writing to cuc_actor in s3, kcgwiki and labtestwiki"

https://gerrit.wikimedia.org/r/798813

Change 798993 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Acquire fresh actor id

https://gerrit.wikimedia.org/r/798993

Change 798817 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@wmf/1.39.0-wmf.12] Acquire fresh actor id

https://gerrit.wikimedia.org/r/798817

Change 798818 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@wmf/1.39.0-wmf.13] Acquire fresh actor id

https://gerrit.wikimedia.org/r/798818

Change 800190 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Revert "Revert "Start writing to cuc_actor in s3, kcgwiki and labtestwiki""

https://gerrit.wikimedia.org/r/800190

Change 798818 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@wmf/1.39.0-wmf.13] Acquire fresh actor id

https://gerrit.wikimedia.org/r/798818

Change 800190 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Revert "Start writing to cuc_actor in s3, kcgwiki and labtestwiki""

https://gerrit.wikimedia.org/r/800190

Mentioned in SAL (#wikimedia-operations) [2022-05-26T20:53:39Z] <brennen@deploy1002> Synchronized php-1.39.0-wmf.13/extensions/CheckUser/src/Hooks.php: Backport: [[gerrit:798818|Acquire fresh actor id (T233004 T309148)]] (duration: 00m 51s)

Mentioned in SAL (#wikimedia-operations) [2022-05-26T20:55:05Z] <brennen@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:800190|Revert "Revert "Start writing to cuc_actor in s3, kcgwiki and labtestwiki"" (T233004)]] (duration: 00m 50s)

Change 800278 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Start writing to cuc_actor everywhere except s4 and s8

https://gerrit.wikimedia.org/r/800278

Change 798817 abandoned by Zabe:

[mediawiki/extensions/CheckUser@wmf/1.39.0-wmf.12] Acquire fresh actor id

Reason:

https://gerrit.wikimedia.org/r/798817

@Zabe On my local machine, running php maintenance/update.php:

Wikimedia\Rdbms\DBQueryError from line 1750 of /var/www/html/w/includes/libs/rdbms/database/Database.php: Error 1: no such column: cuc_actor
Function: MediaWiki\CheckUser\Maintenance\PopulateCucActor::doDBUpdates
Query: SELECT  cuc_id,cuc_user_text  FROM cu_changes    WHERE cuc_actor = 0 AND (cuc_id BETWEEN 5908 AND 6108)  

#0 /var/www/html/w/includes/libs/rdbms/database/Database.php(1734): Wikimedia\Rdbms\Database->getQueryException('no such column:...', 1, 'SELECT  cuc_id,...', 'MediaWiki\\Check...')
#1 /var/www/html/w/includes/libs/rdbms/database/Database.php(1708): Wikimedia\Rdbms\Database->getQueryExceptionAndLog('no such column:...', 1, 'SELECT  cuc_id,...', 'MediaWiki\\Check...')
#2 /var/www/html/w/includes/libs/rdbms/database/Database.php(1209): Wikimedia\Rdbms\Database->reportQueryError('no such column:...', 1, 'SELECT  cuc_id,...', 'MediaWiki\\Check...', false)
#3 /var/www/html/w/includes/libs/rdbms/database/Database.php(1954): Wikimedia\Rdbms\Database->query('SELECT  cuc_id,...', 'MediaWiki\\Check...', 32)
#4 /var/www/html/w/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->select('cu_changes', Array, Array, 'MediaWiki\\Check...')
#5 /var/www/html/w/includes/libs/rdbms/database/DBConnRef.php(277): Wikimedia\Rdbms\DBConnRef->__call('select', Array)
#6 /var/www/html/w/extensions/CheckUser/maintenance/populateCucActor.php(101): Wikimedia\Rdbms\DBConnRef->select('cu_changes', Array, Array, 'MediaWiki\\Check...')
#7 /var/www/html/w/maintenance/includes/LoggedUpdateMaintenance.php(45): MediaWiki\CheckUser\Maintenance\PopulateCucActor->doDBUpdates()
#8 /var/www/html/w/maintenance/update.php(205): LoggedUpdateMaintenance->execute()
#9 /var/www/html/w/maintenance/includes/MaintenanceRunner.php(195): UpdateMediaWiki->execute()
#10 /var/www/html/w/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#11 /var/www/html/w/maintenance/update.php(264): require_once('/var/www/html/w...')
#12 {main}

(I also ran it with --force but got the same error.)

CheckUser version 2.5 (e924e27) 06:20, 1 June 2022.

Change 773650 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Add CheckUserActorMigration service

https://gerrit.wikimedia.org/r/773650

Change 800278 merged by jenkins-bot:

[operations/mediawiki-config@master] Start writing to cuc_actor everywhere except s4 and s8

https://gerrit.wikimedia.org/r/800278

Mentioned in SAL (#wikimedia-operations) [2022-06-01T20:26:32Z] <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:800278|Start writing to cuc_actor everywhere except s4 and s8 (T233004)]] (duration: 03m 01s)

Change 802115 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Revert "Start writing to cuc_actor everywhere except s4 and s8"

https://gerrit.wikimedia.org/r/802115

Change 802115 merged by Urbanecm:

[operations/mediawiki-config@master] Revert "Start writing to cuc_actor everywhere except s4 and s8"

https://gerrit.wikimedia.org/r/802115

Mentioned in SAL (#wikimedia-operations) [2022-06-01T20:58:35Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 5a8e7586bcb0933c96e8294e389c31270edb134e: Revert "Start writing to cuc_actor everywhere except s4 and s8" (T233004) (duration: 00m 32s)

@Zabe On my local machine, running php maintenance/update.php:

Wikimedia\Rdbms\DBQueryError from line 1750 of /var/www/html/w/includes/libs/rdbms/database/Database.php: Error 1: no such column: cuc_actor
Function: MediaWiki\CheckUser\Maintenance\PopulateCucActor::doDBUpdates
Query: SELECT  cuc_id,cuc_user_text  FROM cu_changes    WHERE cuc_actor = 0 AND (cuc_id BETWEEN 5908 AND 6108)  

#0 /var/www/html/w/includes/libs/rdbms/database/Database.php(1734): Wikimedia\Rdbms\Database->getQueryException('no such column:...', 1, 'SELECT  cuc_id,...', 'MediaWiki\\Check...')
#1 /var/www/html/w/includes/libs/rdbms/database/Database.php(1708): Wikimedia\Rdbms\Database->getQueryExceptionAndLog('no such column:...', 1, 'SELECT  cuc_id,...', 'MediaWiki\\Check...')
#2 /var/www/html/w/includes/libs/rdbms/database/Database.php(1209): Wikimedia\Rdbms\Database->reportQueryError('no such column:...', 1, 'SELECT  cuc_id,...', 'MediaWiki\\Check...', false)
#3 /var/www/html/w/includes/libs/rdbms/database/Database.php(1954): Wikimedia\Rdbms\Database->query('SELECT  cuc_id,...', 'MediaWiki\\Check...', 32)
#4 /var/www/html/w/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->select('cu_changes', Array, Array, 'MediaWiki\\Check...')
#5 /var/www/html/w/includes/libs/rdbms/database/DBConnRef.php(277): Wikimedia\Rdbms\DBConnRef->__call('select', Array)
#6 /var/www/html/w/extensions/CheckUser/maintenance/populateCucActor.php(101): Wikimedia\Rdbms\DBConnRef->select('cu_changes', Array, Array, 'MediaWiki\\Check...')
#7 /var/www/html/w/maintenance/includes/LoggedUpdateMaintenance.php(45): MediaWiki\CheckUser\Maintenance\PopulateCucActor->doDBUpdates()
#8 /var/www/html/w/maintenance/update.php(205): LoggedUpdateMaintenance->execute()
#9 /var/www/html/w/maintenance/includes/MaintenanceRunner.php(195): UpdateMediaWiki->execute()
#10 /var/www/html/w/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#11 /var/www/html/w/maintenance/update.php(264): require_once('/var/www/html/w...')
#12 {main}

(I also ran it with --force but got the same error.)

CheckUser version 2.5 (e924e27) 06:20, 1 June 2022.

Huh, I can't reproduce and I also don't understand why it is happening. Which RDBMS are you using?

Change 802637 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Stop writing to cuc_actor on all wikis

https://gerrit.wikimedia.org/r/802637

Change 802637 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop writing to cuc_actor on all wikis

https://gerrit.wikimedia.org/r/802637

Mentioned in SAL (#wikimedia-operations) [2022-06-02T21:25:57Z] <jforrester@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Emergency deploy: [[gerrit:802637|Stop writing to cuc_actor on all wikis (T233004 T309737)]] (duration: 03m 15s)

@Zabe On my local machine, running php maintenance/update.php:
...

Huh, I can't reproduce and I also don't understand why it is happening. Which RDBMS are you using?

sqlite3, using docker.

@Zabe On my local machine, running php maintenance/update.php:
...

Huh, I can't reproduce and I also don't understand why it is happening. Which RDBMS are you using?

sqlite3, using docker.

Yeah, the schema file for sqlite is missing, I will add it.

Change 804482 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/CheckUser@master] Add cuc_actor and cuc_comment_id in sqlite aswell

https://gerrit.wikimedia.org/r/804482

Change 804482 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Add cuc_actor and cuc_comment_id in sqlite aswell

https://gerrit.wikimedia.org/r/804482