Page MenuHomePhabricator

Drop change_tag.ct_tag column in production
Closed, ResolvedPublic

Description

The day has finally come:

  1. ALTERs to run: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/473601/6/maintenance/archives/patch-drop-ct_tag.sql
  2. Where to run those changes: all.dblist
  3. When to run those changes: After deployment of wmf.8, It would be great if we apply it in small and medium wikis first
  4. If the schema change is backwards compatible: No.
  5. If the schema change has been tested already on some of the test/beta wikis: Tested in beta cluster
  6. if the data should be made available on the labs replicas and/or dumps: Yes, data in this table is public (unless the wiki is private which we don't replicate at all)

Execution progress:

s1

s2

s3

s4

s5

s6

s7

s8

wikitech

  • eqiad
  • codfw

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+0 -1
operations/mediawiki-configmaster+7 -7
operations/mediawiki-configmaster+4 -4
operations/mediawiki-configmaster+2 -2
operations/mediawiki-configmaster+2 -2
operations/mediawiki-configmaster+2 -2
operations/mediawiki-configmaster+1 -1
operations/mediawiki-configmaster+1 -1
operations/mediawiki-configmaster+4 -4
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+1 -1
operations/puppetproduction+4 -4
operations/mediawiki-configmaster+5 -5
operations/mediawiki-configmaster+2 -2
operations/mediawiki-configmaster+5 -5
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+2 -2
operations/mediawiki-configmaster+4 -4
operations/mediawiki-configmaster+4 -4
operations/mediawiki-configmaster+3 -3
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+12 -12
operations/mediawiki-configmaster+5 -5
operations/mediawiki-configmaster+2 -2
operations/mediawiki-configmaster+1 -1
operations/mediawiki-configmaster+5 -5
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+5 -5
operations/mediawiki-configmaster+4 -4
operations/mediawiki-configmaster+12 -12
operations/mediawiki-configmaster+2 -2
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+3 -3
operations/mediawiki-configmaster+4 -4
operations/mediawiki-configmaster+4 -4
operations/mediawiki-configmaster+2 -2
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+3 -3
operations/mediawiki-configmaster+5 -5
operations/mediawiki-configmaster+5 -5
operations/mediawiki-configmaster+5 -5
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+6 -6
operations/mediawiki-configmaster+2 -2
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 490535 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/490535

Change 490535 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/490535

Mentioned in SAL (#wikimedia-operations) [2019-02-14T06:10:00Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1087 T210713 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2019-02-14T06:10:16Z] <marostegui> Deploy schema change on db1087 with replication, lag will be generated on labsdb:s8 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-14T07:57:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1087 T210713 (duration: 00m 54s)

Change 490620 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/490620

Change 490620 merged by Marostegui:
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/490620

Change 490801 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/490801

Change 490801 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/490801

Mentioned in SAL (#wikimedia-operations) [2019-02-15T06:17:48Z] <marostegui> Deploy schema change on db1109 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T05:55:22Z] <marostegui> Deploy schema change on s8 primary master (db1071) - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T08:23:45Z] <marostegui> Deploy schema change on s1 codfw master (db2048), lag will be generated on s1 codfw - T210713

s1 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • dbstore1002
  • dbstore1001
  • db1124
  • db1119
  • db1118
  • db1106
  • db1105
  • db1099
  • db1089
  • db1083
  • db1080
  • db1067 T210713#4967984

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:08:13Z] <marostegui> Deploy schema change on dbstore1003:3311 and dbstore1001:3311 - T210713

Change 491218 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/491218

Change 491218 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/491218

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:16:39Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T210713 (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:16:43Z] <marostegui> Deploy schema change on db1099:3311 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:46:47Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T210713 (duration: 00m 46s)

Change 491225 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/491225

Change 491225 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/491225

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:50:39Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:50:45Z] <marostegui> Deploy schema change on db1105:3311 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T10:54:10Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T210713 (duration: 00m 46s)

Change 491409 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/491409

Change 491409 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/491409

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:03Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1106 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:21Z] <marostegui> Deploy schema change on db1106 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:45Z] <marostegui> Deploy schema change on db1106 - this will generate lag on labsdb:s1 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:05:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1106 T210713 (duration: 00m 49s)

Change 491416 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/491416

Change 491416 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/491416

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:50:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1089 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:50:29Z] <marostegui> Deploy schema change on db1089 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T09:29:35Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1089 T210713 (duration: 00m 45s)

Change 491433 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/491433

Change 491433 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/491433

Mentioned in SAL (#wikimedia-operations) [2019-02-19T10:25:34Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1083 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T10:25:43Z] <marostegui> Deploy schema change on db1083 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:12:24Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1083 T210713 (duration: 00m 46s)

Change 491449 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1118

https://gerrit.wikimedia.org/r/491449

Change 491449 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1118

https://gerrit.wikimedia.org/r/491449

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:59:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1118 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:59:55Z] <marostegui> Deploy schema change on db1118 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T13:25:45Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1118 T210713 (duration: 00m 46s)

Change 491682 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/491682

Change 491682 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/491682

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:12:20Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1119 T210713 (duration: 01m 05s)

Change 491689 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/491689

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:44:32Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1119 T210713 (duration: 00m 51s)

Change 491689 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/491689

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:47:46Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 T210713 (duration: 00m 51s)

Mentioned in SAL (#wikimedia-operations) [2019-02-20T07:13:51Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1080 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-20T07:14:05Z] <marostegui> Deploy schema change on s1 primary master (db1067) - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-20T09:33:28Z] <marostegui> Deploy schema change on db2043 (s3 codfw master), lag will be generated on s3 codfw - T210713

db1067 (s1 master) has too much concurrency to let the alter go thru, I will try a few more times before giving up on it and leaving it for when we either failover the master or the DC.

Mentioned in SAL (#wikimedia-operations) [2019-02-20T10:04:06Z] <marostegui> Deploy schema change on dbstore1004:3313 - T210713

s3 eqiad

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • dbstore1002
  • db1124
  • db1123
  • db1095
  • db1077
  • db1075
  • db1078

Mentioned in SAL (#wikimedia-operations) [2019-02-20T10:36:56Z] <marostegui> Deploy schema change on db1095:3313 - T210713

Change 491900 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/491900

Change 491900 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/491900

Mentioned in SAL (#wikimedia-operations) [2019-02-21T06:21:44Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1123 T210713 (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T06:22:08Z] <marostegui> Deploy schema change on db1123 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:08:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1123 T210713 (duration: 00m 55s)

Change 491906 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/491906

Change 491906 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/491906

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:12:54Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1077 T210713 (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:12:57Z] <marostegui> Deploy schema change on db1077 - this will generate lag on labsdb:s3 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:35:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1077 T210713 (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:40:36Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1075 T210713 (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:40:46Z] <marostegui> Deploy schema change on db1075 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T09:29:31Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1075 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T09:29:34Z] <marostegui> Deploy schema change on s3 primary master (db1078) - T210713

Change 491920 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] filtered_tables.txt: Remove change_tag.ct_tag column

https://gerrit.wikimedia.org/r/491920

All the hosts are done except db1067 (s1 master T210713#4967984 ) which I will try a few more times before stalling this until we do a failover.

Change 491920 merged by Marostegui:
[operations/puppet@production] filtered_tables.txt: Remove change_tag.ct_tag column

https://gerrit.wikimedia.org/r/491920

Marostegui changed the task status from Open to Stalled.Feb 26 2019, 8:02 AM

Stalling this until we have failed over s1 master, as it is impossible to alter that host whilst it is active.

DannyS712 changed the task status from Stalled to Open.Nov 14 2019, 6:41 AM
DannyS712 subscribed.

Stalling this until we have failed over s1 master, as it is impossible to alter that host whilst it is active.

Hehe, thanks! I am actually running the schema change already :)

Hehe, thanks! I am actually running the schema change already :)

db1067 has finally been altered:

root@db1067.eqiad.wmnet[enwiki]> ALTER TABLE /*_*/change_tag MODIFY ct_tag_id int unsigned NOT NULL;
Query OK, 92366667 rows affected (53 min 51.55 sec)
Records: 92366667  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]>
root@db1067.eqiad.wmnet[enwiki]> DROP INDEX /*i*/change_tag_rc_tag_nonuniq ON /*_*/change_tag;
Query OK, 0 rows affected (1.35 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]> DROP INDEX /*i*/change_tag_log_tag_nonuniq ON /*_*/change_tag;
Query OK, 0 rows affected (2.65 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]> DROP INDEX /*i*/change_tag_rev_tag_nonuniq ON /*_*/change_tag;
Query OK, 0 rows affected (0.48 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]> DROP INDEX /*i*/change_tag_tag_id ON /*_*/change_tag;
Query OK, 0 rows affected (2.65 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]>
root@db1067.eqiad.wmnet[enwiki]> ALTER TABLE /*_*/change_tag DROP COLUMN ct_tag;
Query OK, 0 rows affected (34 min 49.92 sec)
Records: 0  Duplicates: 0  Warnings: 0
Marostegui updated the task description. (Show Details)