Page MenuHomePhabricator

Deploy patch-gjlw_namespace_text.sql on x1.commonswiki for JsonConfig
Closed, ResolvedPublic

Description

(Updated 2025-02-21 to reflect recommended schema layout for this case.)
T383591's update to JsonConfig requires a single column addition on the globaljsonlinks_wiki table. This is small, and isolated on x1.commonswiki in production.

Initial config steps:

  • Set $wgTrackGlobalJsonLinksNamespace = false; in config so nothing breaks if the patch goes live early
  • Land patch for T383591

DB steps:

  • 1. Apply sql/mysql/patch-gjlw_namespace_text.sql on x1.commonswiki
  • 2. run on x1.commonswiki
  • 3. this can be run at any time, the wiki will have the new column preffed off before patch deployment, until it's confirmed ready
  • 4. schema change *is* backwards compatible, old code will record null column values which will be auto-upgraded later
  • 5. schema change has not yet been tested on test -- to try it, apply on testcommonswiki
  • 6. no sensitive data, ok to include this table and all its columns in replicas and dumps
  • 7. this schema change blocks full rollout of cross-wiki usage tracking display for Charts extension and other Data:.tab pages used via JsonConfig

Final config steps:

  • Set $wgTrackGlobalJsonLinksNamespace = true; in config to start recording data in the new column.

This can be run at any time, and should be very quick, it's a relatively small table and this adds a non-indexed nullable integer and a non-indexed nullable text column, with a change to the unique index that restricts entries. Old entries will default to null in old rows, with a fallback to canonical namespaces for most cases, and will be back-filled by later regular operations. No manual intervention is expected to be needed.

Event Timeline

Change #1118184 had a related patch set uploaded (by Bvibber; author: Bvibber):

[operations/mediawiki-config@master] Pref off use of gjl_namespace_text field until it's deployed

https://gerrit.wikimedia.org/r/1118184

Please follow the process for schema change deployment: https://wikitech.wikimedia.org/wiki/Schema_changes#Workflow_of_a_schema_change

thanks, I've added what I hope is all the info you need. :) Please ping with any questions and comments!

This probably requires SBR in x1 otherwise it breaks replicas.
It is already running SBR temporarily since a week. But better to be double checked just in case.

IIRC we discussed in the charts schema design that the namesapce should be in a group with wiki, and that should become a FK being added to globaljsonlinks. The current schema change will be hard to clean up when namespaces rename on-wiki.

IIRC we discussed in the charts schema design that the namesapce should be in a group with wiki, and that should become a FK being added to globaljsonlinks. The current schema change will be hard to clean up when namespaces rename on-wiki.

As we found during initial implementation it isn't possible to group the namespace text with the wikis: namespaces can vary per-title, as for example the User and User_talk namespaces in Polish localization which vary by the users' gender and must be looked up and localized individually within the TitleFormatter.

GlobalUsage uses the same layout as we do here, and it turns out it does so for a good reason.

Change #1118184 merged by jenkins-bot:

[operations/mediawiki-config@master] Pref off use of gjl_namespace_text field until it's deployed

https://gerrit.wikimedia.org/r/1118184

Mentioned in SAL (#wikimedia-operations) [2025-02-10T21:38:39Z] <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1118184|Pref off use of gjl_namespace_text field until it's deployed (T385917)]], [[gerrit:1118507|GrowthExperiments: End A/B test for Community Updates module (T385338)]]

Mentioned in SAL (#wikimedia-operations) [2025-02-10T21:41:15Z] <urbanecm@deploy2002> urbanecm, cyndywikime, bvibber: Backport for [[gerrit:1118184|Pref off use of gjl_namespace_text field until it's deployed (T385917)]], [[gerrit:1118507|GrowthExperiments: End A/B test for Community Updates module (T385338)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-02-10T21:49:08Z] <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1118184|Pref off use of gjl_namespace_text field until it's deployed (T385917)]], [[gerrit:1118507|GrowthExperiments: End A/B test for Community Updates module (T385338)]] (duration: 10m 28s)

IIRC we discussed in the charts schema design that the namesapce should be in a group with wiki, and that should become a FK being added to globaljsonlinks. The current schema change will be hard to clean up when namespaces rename on-wiki.

As we found during initial implementation it isn't possible to group the namespace text with the wikis: namespaces can vary per-title, as for example the User and User_talk namespaces in Polish localization which vary by the users' gender and must be looked up and localized individually within the TitleFormatter.

This could have been mitigated by adding the text column to the wiki + namespace table but not setting a unique key on ns id.

GlobalUsage uses the same layout as we do here, and it turns out it does so for a good reason.

That table is more than 126GB and causing all sorts of issues for us. It really shouldn't be used as a guide.

It's too late to do the data migration and all of that now but I signed off on a different schema than what it ended up in production.

This could have been mitigated by adding the text column to the wiki + namespace table but not setting a unique key on ns id.

@Ladsgroup Hmm, I suppose we could do that. That'll mean adding an index on the field, making the wikis table bigger, but smaller than the resulting size saved on globaljsonlinks. If you feel this is the better way to go I'll retool the patch; it's best to get it right this time. :)

If it'd be possible, let's go this way to reduce the pain in the long-term.

bvibber renamed this task from Deploy patch-gjl_namespace_text.sql on x1.commonswiki for JsonConfig to Deploy patch-gjlw_namespace_text.sql on x1.commonswiki for JsonConfig.Feb 21 2025, 7:51 PM
bvibber updated the task description. (Show Details)

@Ladsgroup I'm reasonably happy with the state of the updated patch, please go ahead and confirm the updated schema is in line with your recommendations and good to go. Happy to make final tweaks as necessary!

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/JsonConfig/+/1111690

CCiufo-WMF edited projects, added Charts (Sprint 17); removed Charts (Sprint 15).
CCiufo-WMF moved this task from Incoming to Code Review on the Charts (Sprint 17) board.

@Ladsgroup I'm reasonably happy with the state of the updated patch, please go ahead and confirm the updated schema is in line with your recommendations and good to go. Happy to make final tweaks as necessary!

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/JsonConfig/+/1111690

Once the patch is merged, we will deploy the patch.

I have rolled back the train cause patch-gjlw_namespace_text.sql was not applied on mediawikiwiki and apparently deploying https://gerrit.wikimedia.org/r/c/mediawiki/extensions/JsonConfig/+/1111690 this week cause the code to use it. That has led to unbreak now T387843: Unknown column 'gjlw_namespace' in 'where clause'.

Change #1124488 had a related patch set uploaded (by Bvibber; author: Bvibber):

[operations/mediawiki-config@master] Fix typo in wgTrackGlobalJsonLinksNamespaces

https://gerrit.wikimedia.org/r/1124488

Change #1124488 merged by jenkins-bot:

[operations/mediawiki-config@master] Fix typo in wgTrackGlobalJsonLinksNamespaces

https://gerrit.wikimedia.org/r/1124488

Mentioned in SAL (#wikimedia-operations) [2025-03-04T18:28:53Z] <hashar@deploy2002> Started scap sync-world: Backport for [[gerrit:1124488|Fix typo in wgTrackGlobalJsonLinksNamespaces (T387843 T385917)]]

Mentioned in SAL (#wikimedia-operations) [2025-03-04T18:36:09Z] <hashar@deploy2002> hashar, bvibber: Backport for [[gerrit:1124488|Fix typo in wgTrackGlobalJsonLinksNamespaces (T387843 T385917)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-03-04T18:43:45Z] <hashar@deploy2002> Finished scap sync-world: Backport for [[gerrit:1124488|Fix typo in wgTrackGlobalJsonLinksNamespaces (T387843 T385917)]] (duration: 14m 51s)

Once the patch is merged, we will deploy the patch.

Ok, current status is:

  • patch is merged
  • we fixed the config typo that broke it on the old schema
  • confirmed train is fixed and code in old-schema-mode is running on Commons :D
  • we are ready for the schema update :D

Once that's done I can update the config again to switch in use of the new columns.

@bvibber can you confirm this schema change needs to be applied to:

x1.commonswiki
s4. testcommonswiki

Thanks

@bvibber can you confirm this schema change needs to be applied to:

x1.commonswiki
s4. testcommonswiki

Correct.

Mentioned in SAL (#wikimedia-operations) [2025-03-11T10:21:41Z] <marostegui> Deploy schema change on s4 testcommonswiki codfw master with replication dbmaint T385917

Mentioned in SAL (#wikimedia-operations) [2025-03-11T10:22:42Z] <marostegui> Deploy schema change on x1 commonswiki codfw master with replication dbmaint T385917

@bvibber can you confirm this schema change needs to be applied to:

x1.commonswiki
s4. testcommonswiki

Correct.

This has been run on both.

Woohoo, thanks! I'll get on the after config updates. :D

Change #1127976 had a related patch set uploaded (by Bvibber; author: Bvibber):

[operations/mediawiki-config@master] Re-enable wgTrackGlobalJsonLinksNamespaces for JsonConf

https://gerrit.wikimedia.org/r/1127976

Change #1127976 merged by jenkins-bot:

[operations/mediawiki-config@master] Re-enable wgTrackGlobalJsonLinksNamespaces for JsonConf

https://gerrit.wikimedia.org/r/1127976

Mentioned in SAL (#wikimedia-operations) [2025-03-17T20:06:29Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1127677|Enable Vector 2022 on Wikidata (T387154)]], [[gerrit:1127155|Enable Donation banner on Catalan Wikipedia (T387768)]], [[gerrit:1127976|Re-enable wgTrackGlobalJsonLinksNamespaces for JsonConf (T385917)]]

Mentioned in SAL (#wikimedia-operations) [2025-03-17T20:10:40Z] <tgr@deploy2002> bvibber, jdlrobson, tgr: Backport for [[gerrit:1127677|Enable Vector 2022 on Wikidata (T387154)]], [[gerrit:1127155|Enable Donation banner on Catalan Wikipedia (T387768)]], [[gerrit:1127976|Re-enable wgTrackGlobalJsonLinksNamespaces for JsonConf (T385917)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-03-17T20:19:57Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1127677|Enable Vector 2022 on Wikidata (T387154)]], [[gerrit:1127155|Enable Donation banner on Catalan Wikipedia (T387768)]], [[gerrit:1127976|Re-enable wgTrackGlobalJsonLinksNamespaces for JsonConf (T385917)]] (duration: 13m 28s)

I don't know exactly how to test + sign off on this. @bvibber if you believe the work here is complete, feel free to resolve the task.

I don't know exactly how to test + sign off on this. @bvibber if you believe the work here is complete, feel free to resolve the task.

I consider your signing off on T389553 to finish this. <3 :D

Closing.