Page MenuHomePhabricator
Create Task
Maniphest T381599

Migrate current references of text table rows from afl_var_dump
Closed, ResolvedPublic
Actions

Description

afl_var_dump uses BlobStore, so before T362566: Stop growth of text table by storing ES addresses in content table each AbuseFilter hits will add one row to text table. In Wikidata and English Wikipedia there are more than 30 million hits. So we can remove 30 million more rows from text table in addition to T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production).

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add migration script for migrating references of es to afl tablemediawiki/extensions/AbuseFiltermaster+227 -0
Add migration script for migrating references of es to afl tablemediawiki/extensions/WikimediaMaintenancemaster+207 -0
Customize query in gerrit

Related Objects

Event Timeline

Bugreporter created this task.Dec 5 2024, 5:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2024, 5:24 PM
Bugreporter mentioned this in T382778: Optimize text table.Dec 29 2024, 4:26 AM
Bugreporter mentioned this in T381629: Remove rows about file metadata from text table.Feb 17 2025, 11:53 PM
Zabe claimed this task.Feb 18 2025, 3:20 PM
gerritbot added a comment.Feb 19 2025, 2:37 AM

Change #1120579 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/WikimediaMaintenance@master] Add migration script for migrating references of es to afl table

https://gerrit.wikimedia.org/r/1120579

gerritbot added a project: Patch-For-Review.Feb 19 2025, 2:37 AM
gerritbot added a comment.Feb 20 2025, 11:27 PM

Change #1120579 abandoned by Zabe:

[mediawiki/extensions/WikimediaMaintenance@master] Add migration script for migrating references of es to afl table

https://gerrit.wikimedia.org/r/1120579

gerritbot added a comment.Feb 20 2025, 11:29 PM

Change #1121463 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/AbuseFilter@master] Add migration script for migrating references of es to afl table

https://gerrit.wikimedia.org/r/1121463

gerritbot added a comment.Feb 26 2025, 1:39 PM

Change #1121463 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] Add migration script for migrating references of es to afl table

https://gerrit.wikimedia.org/r/1121463

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.19; 2025-03-04).Feb 26 2025, 2:00 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 26 2025, 2:30 PM
Zabe added a comment.Mar 14 2025, 1:29 AM

I ran the script on beta cluster now, let's see if something blows up.

Stashbot added a comment.Mar 14 2025, 9:06 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-14T21:06:44Z] <zabe> zabe@mwmaint2002:~$ mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php testwiki --dump /home/zabe/afl_text_table_dump/testwiki --deletedump /home/zabe/afl_text_table_deletedump/testwiki --sleep 0.3 # T381599

Stashbot added a comment.Mar 14 2025, 9:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-14T21:25:56Z] <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php testwiki --delete /home/zabe/afl_text_table_deletedump/testwiki --sleep 0.3 # T381599

Stashbot added a comment.Mar 17 2025, 11:37 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-17T23:37:43Z] <zabe> zabe@mwmaint2002:~$ cat group0.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php {} --deletedump /home/zabe/afl_text_table_deletedump/{} --dump /home/zabe/afl_text_table_dump/{}" # T381599

Stashbot added a comment.Mar 18 2025, 12:14 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-18T00:14:05Z] <zabe> zabe@mwmaint2002:~$ cat group0.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php {} --delete /home/zabe/afl_text_table_deletedump/{} --sleep 0.3" # T381599

Stashbot added a comment.Mar 26 2025, 1:11 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-26T01:11:55Z] <zabe> zabe@mwmaint1002:~$ cat group1.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php {} --deletedump /home/zabe/afl_text_table_deletedump/{} --dump /home/zabe/afl_text_table_dump/{} --sleep 0.3" # T381599

Zabe added a comment.Mar 26 2025, 2:19 AM
In T381599#10676713, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2025-03-26T01:11:55Z] <zabe> zabe@mwmaint1002:~$ cat group1.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php {} --deletedump /home/zabe/afl_text_table_deletedump/{} --dump /home/zabe/afl_text_table_dump/{} --sleep 0.3" # T381599

I am skipping commonswiki and wikidatawiki here.

Stashbot added a comment.Mar 27 2025, 11:43 PM

Mentioned in SAL (#wikimedia-operations) [2025-03-27T23:43:56Z] <zabe> zabe@mwmaint1002:~$ cat group1.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php {} --delete /home/zabe/afl_text_table_deletedump/{} --sleep 0.3" # T381599

Stashbot added a comment.Apr 3 2025, 12:15 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-03T00:15:36Z] <zabe> zabe@mwmaint1002:~$ cat group2.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php {} --deletedump /home/zabe/afl_text_table_deletedump/{} --dump /home/zabe/afl_text_table_dump/{} --sleep 0.4" # T381599

Zabe added a comment.Apr 3 2025, 12:22 AM
In T381599#10706454, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2025-04-03T00:15:36Z] <zabe> zabe@mwmaint1002:~$ cat group2.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php {} --deletedump /home/zabe/afl_text_table_deletedump/{} --dump /home/zabe/afl_text_table_dump/{} --sleep 0.4" # T381599

skipping enwiki here

Stashbot added a comment.Apr 14 2025, 11:12 PM

Mentioned in SAL (#wikimedia-operations) [2025-04-14T23:12:02Z] <zabe> zabe@mwmaint1002:~$ cat group2.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php {} --delete /home/zabe/afl_text_table_deletedump/{} --sleep 0.3" # T381599

Stashbot added a comment.Apr 28 2025, 11:54 PM

Mentioned in SAL (#wikimedia-operations) [2025-04-28T23:54:07Z] <zabe> zabe@mwmaint1002:~$ mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php enwiki --deletedump /home/zabe/afl_text_table_deletedump/enwiki --dump /home/zabe/afl_text_table_dump/enwiki --sleep 0.5 # T381599

Zabe mentioned this in T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production).Apr 30 2025, 4:01 PM
Stashbot added a comment.May 5 2025, 11:14 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-05T23:14:40Z] <zabe> zabe@mwmaint1002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php enwiki --delete /home/zabe/afl_text_table_deletedump/enwiki --sleep 0.3 # T381599

Stashbot added a comment.Jun 16 2025, 9:31 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-16T09:31:52Z] <zabe> zabe@deploy1003:~$ mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php wikidatawiki --deletedump /home/zabe/afl_text_table_deletedump/wikidatawiki --dump /home/zabe/afl_text_table_dump/wikidatawiki --sleep 0.5 # T381599

Stashbot added a comment.Jun 24 2025, 7:46 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-24T19:46:33Z] <zabe> zabe@deploy1003:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php wikidatawiki --delete /home/zabe/afl_text_table_deletedump/wikidatawiki --sleep 0.5 # T381599

Zabe closed this task as Resolved.Jun 30 2025, 6:40 PM
wikiadmin2023@10.64.16.207(wikidatawiki)> select count(*) from text;
+----------+
| count(*) |
+----------+
|       35 |
+----------+
1 row in set (0.001 sec)

wikiadmin2023@10.64.16.207(wikidatawiki)>
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits