Page MenuHomePhabricator
Create Task
Maniphest T183490

MCR schema migration stage 4: Migrate External Store URLs (wmf production)
Closed, ResolvedPublic
Actions

Assigned To
Zabe
Authored By
daniel
Dec 21 2017, 6:43 PM
Tags
Referenced Files
F58392603: output_group1.txt
Feb 12 2025, 8:02 PM
F58392604: output_group2.txt
Feb 12 2025, 8:02 PM
F57511562: Screenshot 2024-09-14 at 20.24.15.png
Sep 15 2024, 3:26 AM
F57511536: Screenshot 2024-09-14 at 20.11.39.png
Sep 15 2024, 3:12 AM
Subscribers
Aklapper
Amire80
ArielGlenn
aude
Ciencia_Al_Poder
Daimona
daniel
View All 16 Subscribers

Description

We should migrate data stored in the External Store away from the text table: The External Store URL that is contained in the text blob can be written to the cont_address field (possibly with a prefix, to be decided, see External Store Integration). Then the corresponding rows can be deleted from the text table.

Progress:

  • s1
  • s2
  • s3
  • s4
  • s5
  • s6
  • s7
  • s8

The scripts are running in screens on mwmaint2002. Since we are deleting data from the text table, we are generating a dump of the data, where the id of a text table row is associated with the external storage address, in order to provide a backup in case of something going wrong. After a wiki migrated successfully the dump is gziped and moved to stat1009 (since that host has far more disk space).

In case of data corruption: You can find the dump for the text table of abcwiki at stat1009.eqiad.wmnet:/home/zabe/text_table_dump_compressed/abcwiki.gz.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Stop setting wgRevisionSlotsCacheExpiryoperations/mediawiki-configmaster+0 -5
Remove $wgRevisionSlotsCacheExpirymediawiki/coremaster+3 -50
wikidatawiki: Increase revision-slots cache back to defaultoperations/mediawiki-configmaster+0 -1
enwiki and commons: Increase revision-slots cache expiry againoperations/mediawiki-configmaster+0 -2
Increase revision-slots cache expiry back to default for 3 wikisoperations/mediawiki-configmaster+0 -5
Reduce revision-slots cache expiry to 60s on diqwiki and ttwikioperations/mediawiki-configmaster+4 -0
Add script to delete obsolote text table rowsmediawiki/extensions/WikimediaMaintenancemaster+104 -0
Increase revision-slots cache expiry back to default for most wikisoperations/mediawiki-configmaster+7 -2
MCR Stage 4: Migrate references of external storage to content tablemediawiki/coremaster+212 -0
s8: Reduce revision-slots cache expiry to 60 secondsoperations/mediawiki-configmaster+0 -1
s1: Reduce revision-slots cache expiry to 60 secondsoperations/mediawiki-configmaster+0 -1
group0: Increase revision-slots cache expiry back to defaultoperations/mediawiki-configmaster+1 -0
s4: Reduce revision-slots cache expiry to 60 secondsoperations/mediawiki-configmaster+0 -8
s7: Reduce revision-slots cache expiry to 60 secondsoperations/mediawiki-configmaster+0 -15
s2: Reduce revision-slots cache expiry to 60 secondsoperations/mediawiki-configmaster+1 -20
s5: Reduce revision-slots cache expiry to 60 secondsoperations/mediawiki-configmaster+1 -98
s6: Reduce revision-slots cache expiry to 60soperations/mediawiki-configmaster+1 -7
s3: Reduce revision-slots cache expiry to 60 secondsoperations/mediawiki-configmaster+148 -4
Make revision-slots expiry configurablemediawiki/corewmf/1.43.0-wmf.24+50 -3
Make revision-slots expiry configurablemediawiki/coremaster+50 -3
Pin wgRevisionSlotsCacheExpiry to defaultoperations/mediawiki-configmaster+8 -0
migrateESRefToContentTable: Check for maximal not migrated content_idmediawiki/extensions/WikimediaMaintenancemaster+11 -1
migrateESRefToContentTable: Add option to dump tt: -> es: referencemediawiki/extensions/WikimediaMaintenancewmf/1.43.0-wmf.21+22 -0
migrateESRefToContentTable: Add option for not deleting text rowmediawiki/extensions/WikimediaMaintenancewmf/1.43.0-wmf.21+20 -5
migrateESRefToContentTable: Add option to dump tt: -> es: referencemediawiki/extensions/WikimediaMaintenancewmf/1.43.0-wmf.22+22 -0
migrateESRefToContentTable: Add option for not deleting text rowmediawiki/extensions/WikimediaMaintenancewmf/1.43.0-wmf.22+20 -5
migrateESRefToContentTable: Add option to dump tt: -> es: referencemediawiki/extensions/WikimediaMaintenancemaster+22 -0
migrateESRefToContentTable: Add option for not deleting text rowmediawiki/extensions/WikimediaMaintenancemaster+20 -5
Use a bloom filter to detect duplicate content rowsmediawiki/extensions/WikimediaMaintenancemaster+36 -2
Add migration script for migrating references of es to content tablemediawiki/extensions/WikimediaMaintenancemaster+167 -0
Use a BlobStore for storing var dumpsmediawiki/extensions/AbuseFiltermaster+24 -92
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT174043 Deploy Multi-Content Revisions
 StalledNoneT174045 DB schema migration for MCR
 ResolvedZabeT183490 MCR schema migration stage 4: Migrate External Store URLs (wmf production)
 ResolvedNoneT183487 MCR schema migration stage 3: stop using legacy fields
 ResolvedtstarlingT183488 MCR schema migration stage 2: populate new fields
 ResolvedAnomieT183489 MCR schema migration stage 1: Fix Legacy Archive Rows
 ResolvedAnomieT183486 MCR schema migration stage 0: create tables
 ResolvedAnomieT36925 [MCR] create maintenance script for Migration of text from archive table to text table
 ResolvedAnomieT184629 Run maintenance/migrateArchiveText.php on all wikis
 ResolvedAnomieT191307 Run maintenance/populateArchiveRevId.php on all wikis
 ResolvedAnomieT193180 Clean up archive rows with duplicate revision IDs
 ResolvedAnomieT202032 Duplicate ar_rev_id values in several wikis
 ResolvedAnomieT174030 Implement DB schema migration tools for MCR
 ResolveddanielT174028 Finalize database schema for MCR content meta-data
ResolvedAnomieT182678 [MCR] Script for populating empty ar_rev_id fields
 ResolvedAnomieT182682 [MCR] Script(s) for populating new tables (slots, content, content_models, slot_roles)
 Resolved• MarosteguiT190153 DROP unused 'slots' table (WAS: In the slots table, replace slot_inherited with slot_origin)
 ResolvedAnomieT196172 Test schema migration script
 ResolvedtstarlingT197816 Enable MCR migration stage "write both, read old" on live systems
 ResolvedAnomieT197817 Enable MCR migration stage "write both, read old" on testwiki
 ResolvedAnomieT198312 Set the WMF cluster to use the new MCR-only schema
 ResolveddanielT198558 Set testwiki to use the new MCR-only schema
 ResolveddanielT198559 Set Beta Cluster wikis to use the new MCR-only schema
 ResolvedAnomieT198311 Enable MCR migration stage "write both, read new" on commons beta.
 ResolvedAnomieT197818 Enable MCR migration stage "write both, read old" on commons beta.
 ResolveddanielT197619 Introduce SCHEMA_COMPAT_XXX constants to allow more fine grained control over the migration process
 ResolveddanielT198413 Allow multiple slots to be used while still writing to the old as well as the new schema
 ResolveddanielT236735 Allow WikiExporter to function with MCR-only schema
 ResolveddanielT198706 Make BackupDumper, WikiExporter and XmlDumpWriter compliant with the MCR revision retrival mechanism (main slot only)
 ResolvedBPirkleT203424 Replace the WikiExporter backup dump streaming mode with batched queries
 ResolvedNoneT174047 Hide deprecated/unused fields on toolforge replica [MCR]
 ResolvedLadsgroupT362566 Stop growth of text table by storing ES addresses in content table

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Ahoelzl removed a project: Data-Engineering (Q2 2024 October 1st - December 31th).Dec 18 2024, 1:54 PM
Ladsgroup updated the task description. (Show Details)Dec 26 2024, 3:28 PM

We probably should make the script a bit idempotent. I wanted to run it on dewiki to make sure it's properly cleaned up and it basically took forever.

Ladsgroup mentioned this in T382778: Optimize text table.Dec 26 2024, 3:31 PM
gerritbot added a comment.Dec 26 2024, 4:05 PM

Change #1106929 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/WikimediaMaintenance@master] Add script to delete obsolote text row tables

https://gerrit.wikimedia.org/r/1106929

Zabe updated the task description. (Show Details)Jan 9 2025, 7:21 PM
gerritbot added a comment.Jan 24 2025, 9:08 PM

Change #1114060 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Increase revision-slots cache expiry back to default for most wikis

https://gerrit.wikimedia.org/r/1114060

Zabe updated the task description. (Show Details)Jan 25 2025, 1:31 AM
gerritbot added a comment.Jan 27 2025, 2:33 PM

Change #1114060 merged by jenkins-bot:

[operations/mediawiki-config@master] Increase revision-slots cache expiry back to default for most wikis

https://gerrit.wikimedia.org/r/1114060

Stashbot mentioned this in T384614: Change of default license for Wikinews to CC-BY-4.0 and for fawikinews and svwikinews to CC-BY-SA-4.0 on January 30, 2025.Jan 27 2025, 2:40 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T14:40:19Z] <zabe@deploy2002> Started scap sync-world: T384614 T183490

Stashbot added a comment.Jan 27 2025, 2:44 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T14:44:02Z] <zabe@deploy2002> zabe: T384614 T183490 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Jan 27 2025, 2:57 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T14:57:26Z] <zabe@deploy2002> sync-world aborted: T384614 T183490 (duration: 17m 07s)

Ladsgroup mentioned this in T385167: Run data migration script for file migration.Jan 30 2025, 1:12 PM
Zabe updated the task description. (Show Details)Feb 2 2025, 1:31 AM
Zabe updated the task description. (Show Details)Feb 2 2025, 4:04 PM
Zabe added a comment.EditedFeb 2 2025, 6:02 PM

Current progress on remaining sections.

zabe@mwmaint2002:~/text_table_dump$ wc -l *
   535915916 commonswiki
   447725014 enwiki
   776484342 wikidatawiki
  1760125272 total
zabe@mwmaint2002:~/text_table_dump$

This means that enwiki is about 37.0 % done, commonswiki 61.9 % and wikidatawiki 36.2%.

Ladsgroup added a comment.Feb 3 2025, 9:04 PM

Thanks! The commons one is the most important to me since the file and categorylinks migration will take a long time there and I want to start it ASAP. The others should be fine for now.

Ladsgroup added a comment.Feb 6 2025, 11:16 PM

Cross-posting to make sure this is visible: T382778: Optimize text table

gerritbot added a comment.Feb 9 2025, 1:15 PM

Change #1106929 merged by jenkins-bot:

[mediawiki/extensions/WikimediaMaintenance@master] Add script to delete obsolote text table rows

https://gerrit.wikimedia.org/r/1106929

ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.16; 2025-02-11).Feb 9 2025, 2:00 PM
Stashbot added a comment.Feb 11 2025, 12:59 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-11T00:59:06Z] <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php test2wiki --delete /home/zabe/text_table_cleanup/test2wiki # T183490

Stashbot added a comment.Feb 12 2025, 12:12 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-12T00:12:24Z] <zabe> zabe@mwmaint2002:~$ cat /srv/mediawiki-staging/dblists/group1.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php {} --delete /home/zabe/text_table_cleanup/{} --sleep 0.3" # T183490

Zabe added a comment.EditedFeb 12 2025, 12:32 AM
In T183490#10542245, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2025-02-12T00:12:24Z] <zabe> zabe@mwmaint2002:~$ cat /srv/mediawiki-staging/dblists/group1.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php {} --delete /home/zabe/text_table_cleanup/{} --sleep 0.3" # T183490

To be clear. I obviously skipped commonswiki and wikidatawiki here since those are still running the first stage of this migration.

Zabe added a comment.Feb 12 2025, 1:10 AM

Until dewiktionary there was and is no issue, then T386162 appeared. I stopped the script when it was about enwikibooks.

Ladsgroup added a comment.Feb 12 2025, 2:54 PM
In T183490#10542417, @Zabe wrote:

Until dewiktionary there was and is no issue, then T386162 appeared. I stopped the script when it was about enwikibooks.

Let's run a check to see if any other wiki might have been missed during the first run.

Zabe added a comment.EditedFeb 12 2025, 8:02 PM

Running select count(*) from content where content_address like 'tt%'; on group1 and group2 shows that we apparently missed dewiktionary, diqwiki and ttwiki.

output_group2.txt77 KBDownload

output_group1.txt141 KBDownload

gerritbot added a comment.Feb 12 2025, 11:31 PM

Change #1119207 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Reduce revision-slots cache expiry to 60s on diqwiki and ttwiki

https://gerrit.wikimedia.org/r/1119207

gerritbot added a comment.Feb 13 2025, 12:03 AM

Change #1119207 merged by jenkins-bot:

[operations/mediawiki-config@master] Reduce revision-slots cache expiry to 60s on diqwiki and ttwiki

https://gerrit.wikimedia.org/r/1119207

Stashbot added a comment.Feb 13 2025, 12:04 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-13T00:04:43Z] <zabe@deploy2002> Started scap sync-world: Backport for [[gerrit:1119207|Reduce revision-slots cache expiry to 60s on diqwiki and ttwiki (T183490)]]

Stashbot added a comment.Feb 13 2025, 12:07 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-13T00:07:41Z] <zabe@deploy2002> zabe: Backport for [[gerrit:1119207|Reduce revision-slots cache expiry to 60s on diqwiki and ttwiki (T183490)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Feb 13 2025, 12:15 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-13T00:15:22Z] <zabe@deploy2002> Finished scap sync-world: Backport for [[gerrit:1119207|Reduce revision-slots cache expiry to 60s on diqwiki and ttwiki (T183490)]] (duration: 10m 39s)

Stashbot added a comment.Feb 13 2025, 10:01 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-13T22:01:58Z] <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTable.php diqwiki --skip /home/zabe/text_table_cleanup/diqwiki --dump /home/zabe/text_table_dump/diqwiki --sleep 0.5 --start 318769 # T183490

Stashbot added a comment.Feb 13 2025, 10:38 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-13T22:38:52Z] <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTable.php ttwiki --skip /home/zabe/text_table_cleanup/ttwiki --dump /home/zabe/text_table_dump/ttwiki --sleep 0.5 --start 867501 # T183490

Ladsgroup added a comment.Feb 14 2025, 2:27 PM

diqwiki is also s3 and sits between dewiktionary and enwikibooks. So it probably have also messed up but it's not showing up I guess because it's not really visited? How big is the deletion file for diqwiki?

Zabe added a comment.Feb 14 2025, 4:34 PM
In T183490#10552534, @Ladsgroup wrote:

diqwiki is also s3 and sits between dewiktionary and enwikibooks. So it probably have also messed up but it's not showing up I guess because it's not really visited? How big is the deletion file for diqwiki?

I only ran the stage 2 script on group1 wikis yet and diqwiki is group2, so it shuold be fine.

Zabe added a comment.Feb 14 2025, 4:36 PM
In T183490#10552974, @Zabe wrote:
In T183490#10552534, @Ladsgroup wrote:

diqwiki is also s3 and sits between dewiktionary and enwikibooks. So it probably have also messed up but it's not showing up I guess because it's not really visited? How big is the deletion file for diqwiki?

I only ran the stage 2 script on group1 wikis yet and diqwiki is group2, so it shuold be fine.

Looks good (I did run the stage 1 script in the meantime).

wikiadmin2023@10.192.16.41(diqwiki)> select count(*) from content where content_address like 'tt%';
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.627 sec)

wikiadmin2023@10.192.16.41(diqwiki)>
Ladsgroup added a comment.Feb 14 2025, 4:36 PM

\o/

Stashbot added a comment.Feb 17 2025, 5:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-17T17:25:02Z] <zabe> zabe@mwmaint2002:~$ cat /home/zabe/group2.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php {} --delete /home/zabe/text_table_cleanup/{} --sleep 0.3" # T183490

Zabe added a comment.Feb 18 2025, 3:17 PM
In T183490#10557554, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2025-02-17T17:25:02Z] <zabe> zabe@mwmaint2002:~$ cat /home/zabe/group2.dblist | xargs -I{} bash -c "echo {}; mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php {} --delete /home/zabe/text_table_cleanup/{} --sleep 0.3" # T183490

This finished without issues (I obviously skipped enwiki).

gerritbot added a comment.Feb 19 2025, 1:52 AM

Change #1120689 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Increase revision-slots cache expiry back to default for 3 wikis

https://gerrit.wikimedia.org/r/1120689

gerritbot added a comment.Feb 19 2025, 1:55 AM

Change #1120689 merged by jenkins-bot:

[operations/mediawiki-config@master] Increase revision-slots cache expiry back to default for 3 wikis

https://gerrit.wikimedia.org/r/1120689

Stashbot added a comment.Feb 19 2025, 1:56 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-19T01:56:56Z] <zabe@deploy2002> Started scap sync-world: Backport for [[gerrit:1120690|Activate satwiktionary (T386619)]], [[gerrit:1120689|Increase revision-slots cache expiry back to default for 3 wikis (T183490)]]

Stashbot mentioned this in T386619: Create Wiktionary Santali.Feb 19 2025, 1:56 AM
Stashbot added a comment.Feb 19 2025, 1:59 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-19T01:59:56Z] <zabe@deploy2002> zabe: Backport for [[gerrit:1120690|Activate satwiktionary (T386619)]], [[gerrit:1120689|Increase revision-slots cache expiry back to default for 3 wikis (T183490)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Feb 19 2025, 2:07 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-19T02:07:52Z] <zabe@deploy2002> Finished scap sync-world: Backport for [[gerrit:1120690|Activate satwiktionary (T386619)]], [[gerrit:1120689|Increase revision-slots cache expiry back to default for 3 wikis (T183490)]] (duration: 10m 55s)

Zabe added a comment.Feb 27 2025, 12:45 AM

commonswiki ~92.5%
enwiki ~56.3%
wikidatawiki ~47.7%

Zabe mentioned this in T387604: PHP Warning: Invalid argument supplied for foreach().Mar 1 2025, 12:02 AM
Zabe added a comment.EditedMar 6 2025, 12:32 AM
mysql:research@dbstore1007.eqiad.wmnet [commonswiki]> select count(*) from content where content_address like 'tt%';
+----------+
| count(*) |
+----------+
|      282 |
+----------+
1 row in set (17 min 39.538 sec)

mysql:research@dbstore1007.eqiad.wmnet [commonswiki]>
Stashbot added a comment.Mar 6 2025, 12:33 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-06T00:33:47Z] <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php commonswiki --delete /home/zabe/text_table_cleanup/commonswiki --sleep 0.5 # T183490

Zabe updated the task description. (Show Details)Mar 6 2025, 9:44 AM
zdev mentioned this in T28741: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys).Mar 14 2025, 1:28 PM
Zabe added a comment.Mar 18 2025, 1:45 AM

enwiki ~70.7%
wikidatawiki ~57.5%

xcollazo mentioned this in T389199: Fix a performance regression affecting wikibase dumps when using mediawiki analytics replica of s8 - dbstore1009.Mar 18 2025, 4:01 PM
Zabe added a comment.Mar 23 2025, 2:14 AM

Moved the scripts that are still running (enwiki and wikidatawiki) back to mwmaint1002 due to the dc switchover.

Zabe added a comment.Mar 31 2025, 9:25 PM

enwiki ~79.6%
wikidatawiki ~62.6%

Zabe mentioned this in T392834: mwmaint1002 is out of disk space.Apr 28 2025, 11:34 PM
gerritbot added a comment.Apr 29 2025, 12:12 AM

Change #1139577 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] enwiki and commons: Increase revision-slots cache expiry again

https://gerrit.wikimedia.org/r/1139577

gerritbot added a comment.Apr 29 2025, 11:51 PM

Change #1139577 merged by jenkins-bot:

[operations/mediawiki-config@master] enwiki and commons: Increase revision-slots cache expiry again

https://gerrit.wikimedia.org/r/1139577

Stashbot added a comment.Apr 29 2025, 11:51 PM

Mentioned in SAL (#wikimedia-operations) [2025-04-29T23:51:53Z] <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1139577|enwiki and commons: Increase revision-slots cache expiry again (T183490)]]

Stashbot added a comment.Apr 29 2025, 11:58 PM

Mentioned in SAL (#wikimedia-operations) [2025-04-29T23:58:43Z] <zabe@deploy1003> zabe: Backport for [[gerrit:1139577|enwiki and commons: Increase revision-slots cache expiry again (T183490)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Apr 30 2025, 12:05 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-30T00:05:39Z] <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1139577|enwiki and commons: Increase revision-slots cache expiry again (T183490)]] (duration: 13m 45s)

Ladsgroup added a comment.Apr 30 2025, 11:22 AM

Is it finally done in s1/s8?

Zabe added a comment.Apr 30 2025, 4:01 PM

Stage 2 still has to run on s1, but that should take less than a day (I'll do it after stage 1 of T381599 is finished on s1).

On s8, about 500 million rows still have to be migrated.

Ladsgroup added a comment.May 1 2025, 6:52 AM

Ah thanks!

Zabe mentioned this in T393237: Some en.wikipedia pageviews fatal "RevisionAccessException: Failed to load data blob from {address} for revision {revision}.".May 3 2025, 6:32 PM
Zabe updated the task description. (Show Details)May 3 2025, 6:55 PM

enwiki should be done besides T393237.

Maintenance_bot removed a project: Patch-For-Review.May 3 2025, 7:30 PM
Zabe added a comment.Jun 15 2025, 5:25 PM
This comment was removed by Zabe.
Zabe added a comment.Jun 15 2025, 8:52 PM
MariaDB [wikidatawiki]> select count(*) from content where content_address  like 'tt%';
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (48 min 14.734 sec)

MariaDB [wikidatawiki]>
Ladsgroup added a comment.Jun 16 2025, 7:06 AM

\o/

Stashbot added a comment.Jun 16 2025, 8:51 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-16T08:51:37Z] <zabe> zabe@deploy1003:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php wikidatawiki --delete /home/zabe/text_table_cleanup/wikidatawiki --sleep 0.5 # T183490

Zabe updated the task description. (Show Details)Jun 16 2025, 9:18 AM
Zabe closed this task as Resolved.Jun 16 2025, 9:21 AM
Ladsgroup awarded a token.Jun 16 2025, 9:38 AM
gerritbot added a comment.Jun 16 2025, 9:39 AM

Change #1159388 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] wikidatawiki: Increase revision-slots cache back to default

https://gerrit.wikimedia.org/r/1159388

gerritbot added a project: Patch-For-Review.Jun 16 2025, 9:39 AM
gerritbot added a comment.Jun 16 2025, 9:44 AM

Change #1159388 merged by jenkins-bot:

[operations/mediawiki-config@master] wikidatawiki: Increase revision-slots cache back to default

https://gerrit.wikimedia.org/r/1159388

Stashbot mentioned this in T299947: Normalize pagelinks table.Jun 16 2025, 9:45 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-16T09:45:07Z] <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1159388|wikidatawiki: Increase revision-slots cache back to default (T183490)]], [[gerrit:1158804|Stop setting $wgPageLinksSchemaMigrationStage (T299947)]]

Stashbot added a comment.Jun 16 2025, 9:47 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-16T09:47:00Z] <zabe@deploy1003> zabe: Backport for [[gerrit:1159388|wikidatawiki: Increase revision-slots cache back to default (T183490)]], [[gerrit:1158804|Stop setting $wgPageLinksSchemaMigrationStage (T299947)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Jun 16 2025, 9:57 AM

Mentioned in SAL (#wikimedia-operations) [2025-06-16T09:57:54Z] <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1159388|wikidatawiki: Increase revision-slots cache back to default (T183490)]], [[gerrit:1158804|Stop setting $wgPageLinksSchemaMigrationStage (T299947)]] (duration: 12m 46s)

gerritbot added a comment.Jun 16 2025, 10:27 AM

Change #1159402 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/core@master] Remove $wgRevisionSlotsCacheExpiry

https://gerrit.wikimedia.org/r/1159402

gerritbot added a comment.Jun 16 2025, 7:35 PM

Change #1159552 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Stop setting wgRevisionSlotsCacheExpiry

https://gerrit.wikimedia.org/r/1159552

gerritbot added a comment.Jun 17 2025, 10:20 AM

Change #1159402 merged by jenkins-bot:

[mediawiki/core@master] Remove $wgRevisionSlotsCacheExpiry

https://gerrit.wikimedia.org/r/1159402

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.7; 2025-06-24).Jun 17 2025, 11:00 AM
gerritbot added a comment.Jun 24 2025, 7:47 PM

Change #1159552 merged by jenkins-bot:

[operations/mediawiki-config@master] Stop setting wgRevisionSlotsCacheExpiry

https://gerrit.wikimedia.org/r/1159552

Stashbot added a comment.Jun 24 2025, 7:47 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-24T19:47:57Z] <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1159552|Stop setting wgRevisionSlotsCacheExpiry (T183490)]]

Stashbot added a comment.Jun 24 2025, 7:50 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-24T19:50:16Z] <zabe@deploy1003> zabe: Backport for [[gerrit:1159552|Stop setting wgRevisionSlotsCacheExpiry (T183490)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Jun 24 2025, 7:59 PM

Mentioned in SAL (#wikimedia-operations) [2025-06-24T19:59:25Z] <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1159552|Stop setting wgRevisionSlotsCacheExpiry (T183490)]] (duration: 11m 28s)

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2025, 8:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits