Page MenuHomePhabricator

Add namespace column and index to table
Closed, ResolvedPublic

Description

The goal of this task is to define all of the steps and code required for both production and open source release of a database change which adds a column to an existing table, populates the new column with data from other columns and also adds code to the recording function that adds new records which now include the new columns data. In this case, we are adding namespace info to an existing table as a separate column in the Linter table.

The Linter table is used for a variety of reports and allows results record sets to be filtered based on a number of criteria including namespace. The current Linter table does not contain a separate column for each records namespace component of the pages title and thus selects cannot reduce the result set based on that criteria and have to loop through a much larger record set to find those records with the appropriate namespace.

The process of making this type of database change requires a DBA and a developer to cooperate and coordinate their activities and understand the time it takes for some operations to complete and propagate, and other issues such as the ability roll back, UI and new code dependencies and dual mode operation based on feature flag enablement and other minor details. It is hoped this task will touch on and provide a complete guide to developers wishing to accomplish this type of comprehensive task

Details

SubjectRepoBranchLines +/-
operations/mediawiki-configmaster+2 -6
operations/mediawiki-configmaster+2 -0
operations/mediawiki-configmaster+12 -0
mediawiki/extensions/Lintermaster+13 -2
mediawiki/extensions/Lintermaster+2 -2
operations/mediawiki-configmaster+2 -6
operations/mediawiki-configmaster+2 -0
operations/mediawiki-configmaster+12 -0
operations/mediawiki-configmaster+10 -0
operations/mediawiki-configmaster+1 -1
operations/mediawiki-configmaster+5 -0
mediawiki/extensions/Lintermaster+80 -0
mediawiki/extensions/Lintermaster+204 -2
mediawiki/coremaster+106 -0
mediawiki/extensions/Lintermaster+51 -10
mediawiki/extensions/Lintermaster+189 -3
Show related patches Customize query in gerrit

Event Timeline

@Sbailey The ContentTranslation tag in this ticket - was this added intentionally?

It was unintentional, sorry about that,

Shannon

@Sbailey: If I get it right, this is a Documentation task, thus adding tag so this task could be found.
In my understanding, https://wikitech.wikimedia.org/wiki/Schema_changes and https://wikitech.wikimedia.org/wiki/Creating_new_tables cover this mostly.
You may want to add DBA for additional input for specific aspects.

Change 755817 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/extensions/Linter@master] WIP Add namespace column and new index to Linter table

https://gerrit.wikimedia.org/r/755817

Change 756101 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/extensions/Linter@master] WIP Add namespace column and new index to Linter table - part 2

https://gerrit.wikimedia.org/r/756101

So in the process of creating a set of patches that add a column for part 1, start populating the new column when new records are written in part 2, will do a full pass through existing records migrating data derived form data in another column to populate the new column completely in part 3, then add code to utilize the new column for search/reporting in part 4, then do a complete refactor of old and new code, removing old code and moving new code around as needed and removing config variable enablements, etc.

Change 757117 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/extensions/Linter@master] WIP Add namespace column and new index to Linter table - part 3

https://gerrit.wikimedia.org/r/757117

Change 757527 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/extensions/Linter@master] WIP Add namespace column and new index to Linter table - part 4

https://gerrit.wikimedia.org/r/757527

Change 755817 merged by jenkins-bot:

[mediawiki/extensions/Linter@master] Add namespace column and new index to Linter table - part 1

https://gerrit.wikimedia.org/r/755817

Change 756101 merged by jenkins-bot:

[mediawiki/extensions/Linter@master] Add namespace column and new index to Linter table - part 2

https://gerrit.wikimedia.org/r/756101

Change 832340 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/core@master] WIP maintenance script migrates namespace from page to linter table

https://gerrit.wikimedia.org/r/832340

Change 833080 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/extensions/Linter@master] Enable Linter write code of fields namespace and tag and template

https://gerrit.wikimedia.org/r/833080

Change 832340 abandoned by Sbailey:

[mediawiki/core@master] WIP maintenance script migrates namespace from page to linter table

Reason:

The Linter migrateNamespace code is now located in Linter/maintenance

https://gerrit.wikimedia.org/r/832340

Change 858388 had a related patch set uploaded (by Sbailey; author: Sbailey):

[mediawiki/extensions/Linter@master] WIP maintenance script to run linter extension namespace migration

https://gerrit.wikimedia.org/r/858388

Change 757117 merged by jenkins-bot:

[mediawiki/extensions/Linter@master] Phase 3 migrate code for namespace column add to Linter table

https://gerrit.wikimedia.org/r/757117

Change 858388 merged by jenkins-bot:

[mediawiki/extensions/Linter@master] maintenance script to run linter extension namespace migration

https://gerrit.wikimedia.org/r/858388

Change 866518 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] enable migrate namespace function on beta

https://gerrit.wikimedia.org/r/866518

Change 866518 merged by jenkins-bot:

[operations/mediawiki-config@master] enable migrate namespace function on beta

https://gerrit.wikimedia.org/r/866518

Change 867724 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] enable Linter extension maintNamespace.php in Beta

https://gerrit.wikimedia.org/r/867724

Change 867724 merged by jenkins-bot:

[operations/mediawiki-config@master] enable Linter extension maintNamespace.php in Beta

https://gerrit.wikimedia.org/r/867724

Change 833080 abandoned by Sbailey:

[mediawiki/extensions/Linter@master] Enable Linter write code of fields namespace and tag and template

Reason:

replaced with other patch

https://gerrit.wikimedia.org/r/833080

Change 833080 restored by Sbailey:

[mediawiki/extensions/Linter@master] Enable Linter write code of fields namespace and tag and template

https://gerrit.wikimedia.org/r/833080

Change 880989 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] Enable Linter write namespace tag and template using core config

https://gerrit.wikimedia.org/r/880989

Change 880989 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Linter write namespace tag and template using core config

https://gerrit.wikimedia.org/r/880989

Mentioned in SAL (#wikimedia-operations) [2023-01-23T14:52:11Z] <taavi@deploy1002> Started scap: Backport for [[gerrit:880989|Enable Linter write namespace tag and template using core config (T299612)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-23T14:53:47Z] <taavi@deploy1002> taavi and sbailey: Backport for [[gerrit:880989|Enable Linter write namespace tag and template using core config (T299612)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-23T15:00:08Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:880989|Enable Linter write namespace tag and template using core config (T299612)]] (duration: 07m 56s)

Change 884090 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] Enable Linter write namespace, tag and template from core, group0

https://gerrit.wikimedia.org/r/884090

Mentioned in SAL (#wikimedia-operations) [2023-01-23T15:00:08Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:880989|Enable Linter write namespace tag and template using core config (T299612)]] (duration: 07m 56s)

For the record, this got reverted because of DB errors (see #wikimedia-operations). I’m hoping those are fixed now.

The drift report didn't find any further missing schema changes, so this should be good to go (but gradual rollout is always encouraged)

Change 884090 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Linter write namespace, tag and template from core, group0

https://gerrit.wikimedia.org/r/884090

Mentioned in SAL (#wikimedia-operations) [2023-01-30T14:35:03Z] <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:884090|Enable Linter write namespace, tag and template from core, group0 (T299612)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-30T14:36:44Z] <lucaswerkmeister-wmde@deploy1002> lucaswerkmeister-wmde and sbailey: Backport for [[gerrit:884090|Enable Linter write namespace, tag and template from core, group0 (T299612)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-30T14:46:14Z] <lucaswerkmeister-wmde@deploy1002> Finished scap: Backport for [[gerrit:884090|Enable Linter write namespace, tag and template from core, group0 (T299612)]] (duration: 11m 11s)

Change 885046 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] Enable Linter write namespace, tag and template for group0 and group1

https://gerrit.wikimedia.org/r/885046

Change 885046 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Linter write namespace, tag and template for group0 and group1

https://gerrit.wikimedia.org/r/885046

Mentioned in SAL (#wikimedia-operations) [2023-01-31T21:04:27Z] <kindrobot@deploy1002> Started scap: Backport for [[gerrit:885046|Enable Linter write namespace, tag and template for group0 and group1 (T299612)]]

Mentioned in SAL (#wikimedia-operations) [2023-01-31T21:06:17Z] <kindrobot@deploy1002> sbailey and kindrobot: Backport for [[gerrit:885046|Enable Linter write namespace, tag and template for group0 and group1 (T299612)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-01-31T21:17:48Z] <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:885046|Enable Linter write namespace, tag and template for group0 and group1 (T299612)]] (duration: 13m 20s)

Change 885852 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] Enable Linter write namespace, tag and template for all wikis

https://gerrit.wikimedia.org/r/885852

Change 885852 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Linter write namespace, tag and template for all wikis

https://gerrit.wikimedia.org/r/885852

Mentioned in SAL (#wikimedia-operations) [2023-02-01T21:56:08Z] <kindrobot@deploy1002> Started scap: Backport for [[gerrit:885852|Enable Linter write namespace, tag and template for all wikis (T299612)]]

Mentioned in SAL (#wikimedia-operations) [2023-02-01T21:57:57Z] <kindrobot@deploy1002> kindrobot and sbailey: Backport for [[gerrit:885852|Enable Linter write namespace, tag and template for all wikis (T299612)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-01T22:14:22Z] <kindrobot@deploy1002> Finished scap: Backport for [[gerrit:885852|Enable Linter write namespace, tag and template for all wikis (T299612)]] (duration: 18m 14s)

Change 833080 abandoned by Sbailey:

[mediawiki/extensions/Linter@master] Enable Linter write code of fields namespace and tag and template

Reason:

https://gerrit.wikimedia.org/r/833080

Change 757527 merged by jenkins-bot:

[mediawiki/extensions/Linter@master] Use new linter_namespace column and index in the Linter table

https://gerrit.wikimedia.org/r/757527

Field added, maintenance migrate run and results verified in Beta and production.

Change 893811 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] Enable new Linter UI for namespace, tag and template for all wikis

https://gerrit.wikimedia.org/r/893811

Change 893811 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable new Linter UI for namespace, tag and template for group0 wikis

https://gerrit.wikimedia.org/r/893811

Mentioned in SAL (#wikimedia-operations) [2023-03-06T21:02:50Z] <zabe@deploy2002> Started scap: Backport for [[gerrit:893811|Enable new Linter UI for namespace, tag and template for group0 wikis (T299612)]]

Mentioned in SAL (#wikimedia-operations) [2023-03-06T21:04:33Z] <zabe@deploy2002> zabe and sbailey: Backport for [[gerrit:893811|Enable new Linter UI for namespace, tag and template for group0 wikis (T299612)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-03-06T21:19:50Z] <zabe@deploy2002> Finished scap: Backport for [[gerrit:893811|Enable new Linter UI for namespace, tag and template for group0 wikis (T299612)]] (duration: 16m 59s)

Change 894733 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] Enable new Linter UI for namespace, tag and template for group1 wikis

https://gerrit.wikimedia.org/r/894733

Change 894733 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable new Linter UI for namespace, tag and template for group1 wikis

https://gerrit.wikimedia.org/r/894733

Mentioned in SAL (#wikimedia-operations) [2023-03-07T21:23:56Z] <samtar@deploy2002> Started scap: Backport for [[gerrit:894733|Enable new Linter UI for namespace, tag and template for group1 wikis (T299612)]]

Mentioned in SAL (#wikimedia-operations) [2023-03-07T21:25:33Z] <samtar@deploy2002> sbailey and samtar: Backport for [[gerrit:894733|Enable new Linter UI for namespace, tag and template for group1 wikis (T299612)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-03-07T21:33:08Z] <samtar@deploy2002> Finished scap: Backport for [[gerrit:894733|Enable new Linter UI for namespace, tag and template for group1 wikis (T299612)]] (duration: 09m 11s)

Change 895833 had a related patch set uploaded (by Sbailey; author: Sbailey):

[operations/mediawiki-config@master] Enable new Linter UI for namespace, tag and template for all wikis

https://gerrit.wikimedia.org/r/895833

Change 895833 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable new Linter UI for namespace, tag and template for all wikis

https://gerrit.wikimedia.org/r/895833

Mentioned in SAL (#wikimedia-operations) [2023-03-08T21:57:27Z] <kindrobot@deploy2002> Started scap: Backport for [[gerrit:895833|Enable new Linter UI for namespace, tag and template for all wikis (T299612)]]

Mentioned in SAL (#wikimedia-operations) [2023-03-08T21:59:09Z] <kindrobot@deploy2002> sbailey and kindrobot: Backport for [[gerrit:895833|Enable new Linter UI for namespace, tag and template for all wikis (T299612)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-03-08T22:07:03Z] <kindrobot@deploy2002> Finished scap: Backport for [[gerrit:895833|Enable new Linter UI for namespace, tag and template for all wikis (T299612)]] (duration: 09m 36s)