Page MenuHomePhabricator

Enable maintenance Linter data migration scripts for namespace and tag and template
Closed, ResolvedPublic

Description

Namespace and tag and template data migration scripts in linter Database.php are dark launched and require two config variables to be enabled for the scripts to be run.

As the enable variables for maintenance scripts was somewhat duplicitous with the existing config variables that enable the write code, the maintenance data migration scripts should use the same config variables.

They have been extensively reviewed and optimized for safe execution in production, and have been tested by unit tests and run in Beta against all Beta wikis.

In order to run in production, these config variables must be enabled so they can be manually run using a command line such as:
$ mwscriptwikiset extensions/Linter/maintenance/migrateNamespace.php group0.dblist --sleep 1 --batch-size 2000
or
$ mwscriptwikiset extensions/Linter/maintenance/migrateTagTemplate.php group0.dblist --sleep 1 --batch-size 2000

In this example, once group0 is complete additional groups can be run until all wikis have been processed. If there is overlap in groupX.dblist, wikis that have already had their data migrated, will be skipped over automatically. If a script is interrupted, it can be safely rerun, and may need --force option added.

In testing on Beta, a wiki with 120,000 linter records took about five minute to process for each data migration script, so for very large wikis, it may take an hour or more to process all records.
The sleep duration and batch-size should be well chosen for very large wikis to enable replication/syncing to occur between batches.
It might be wise to manually run each maintenance script using mwscript on very large wikis independently instead of running it as part of a group using mwscriptwikiset.

Event Timeline

Change 888111 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] Enable Linter migration scripts for namespace and tag and template

https://gerrit.wikimedia.org/r/888111

Change 888111 abandoned by Sbailey:

[operations/mediawiki-config@master] Enable Linter migration scripts for namespace and tag and template

Reason:

Instead of creating these new variables, switching dependency to existing write config variables.

https://gerrit.wikimedia.org/r/888111

Change 889170 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/extensions/Linter@master] Change linter maintenance scripts to use existing config varaibles

https://gerrit.wikimedia.org/r/889170

Change 889170 merged by jenkins-bot:

[mediawiki/extensions/Linter@master] Change linter maintenance scripts to use existing config varaibles

https://gerrit.wikimedia.org/r/889170

Change 889220 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/extensions/Linter@wmf/1.40.0-wmf.23] Change linter maintenance scripts to use existing config varaibles

https://gerrit.wikimedia.org/r/889220

Change 889220 merged by jenkins-bot:

[mediawiki/extensions/Linter@wmf/1.40.0-wmf.23] Change linter maintenance scripts to use existing config varaibles

https://gerrit.wikimedia.org/r/889220

Mentioned in SAL (#wikimedia-operations) [2023-02-14T21:39:48Z] <dancy@deploy1002> Started scap: Backport for [[gerrit:889220|Change linter maintenance scripts to use existing config varaibles (T329342)]]

Mentioned in SAL (#wikimedia-operations) [2023-02-14T21:41:34Z] <dancy@deploy1002> dancy and sbailey: Backport for [[gerrit:889220|Change linter maintenance scripts to use existing config varaibles (T329342)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-14T21:47:32Z] <dancy@deploy1002> Finished scap: Backport for [[gerrit:889220|Change linter maintenance scripts to use existing config varaibles (T329342)]] (duration: 07m 44s)

Change 889608 had a related patch set uploaded (by Ladsgroup; author: Sbailey):

[mediawiki/extensions/Linter@wmf/1.40.0-wmf.22] Change linter maintenance scripts to use existing config varaibles

https://gerrit.wikimedia.org/r/889608

Change 889608 merged by jenkins-bot:

[mediawiki/extensions/Linter@wmf/1.40.0-wmf.22] Change linter maintenance scripts to use existing config varaibles

https://gerrit.wikimedia.org/r/889608

Mentioned in SAL (#wikimedia-operations) [2023-02-15T23:06:55Z] <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:889608|Change linter maintenance scripts to use existing config varaibles (T329342)]]

Mentioned in SAL (#wikimedia-operations) [2023-02-15T23:08:46Z] <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:889608|Change linter maintenance scripts to use existing config varaibles (T329342)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-15T23:15:07Z] <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:889608|Change linter maintenance scripts to use existing config varaibles (T329342)]] (duration: 08m 12s)