Page MenuHomePhabricator

Drop change_tag.ct_tag column in production
Closed, ResolvedPublic

Description

The day has finally come:

  1. ALTERs to run: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/473601/6/maintenance/archives/patch-drop-ct_tag.sql
  2. Where to run those changes: all.dblist
  3. When to run those changes: After deployment of wmf.8, It would be great if we apply it in small and medium wikis first
  4. If the schema change is backwards compatible: No.
  5. If the schema change has been tested already on some of the test/beta wikis: Tested in beta cluster
  6. if the data should be made available on the labs replicas and/or dumps: Yes, data in this table is public (unless the wiki is private which we don't replicate at all)

Execution progress:

s1

s2

s3

s4

s5

s6

s7

s8

wikitech

  • eqiad
  • codfw

Details

Related Gerrit Patches:
operations/puppet : productionfiltered_tables.txt: Remove change_tag.ct_tag column
operations/mediawiki-config : masterdb-eqiad.php: Depool db1077
operations/mediawiki-config : masterdb-eqiad.php: Depool db1123
operations/mediawiki-config : masterdb-eqiad.php: Depool db1080
operations/mediawiki-config : masterdb-eqiad.php: Depool db1119
operations/mediawiki-config : masterdb-eqiad.php: Depool db1118
operations/mediawiki-config : masterdb-eqiad.php: Depool db1083
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1106
operations/mediawiki-config : masterdb-eqiad.php: Depool db1105:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1099:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1109
operations/puppet : productiondbproxy1011: Depool labsdb1009
operations/mediawiki-config : masterdb-eqiad.php: Depool db1087
operations/mediawiki-config : masterdb-eqiad.php: Depool db1104
operations/mediawiki-config : masterdb-eqiad.php: Depool db1092
operations/mediawiki-config : masterdb-eqiad.php: Depool db1101:3318
operations/mediawiki-config : masterdb-eqiad.php: Depool db1099:3318
operations/mediawiki-config : masterdb-eqiad.php: Depool db1079
operations/mediawiki-config : masterdb-eqiad.php: Depool db1090:3317
operations/mediawiki-config : masterdb-eqiad.php: Depool db1086
operations/mediawiki-config : masterdb-eqiad.php: Depool db1094
operations/mediawiki-config : masterdb-eqiad.php: Depool db1098:3317
operations/mediawiki-config : masterdb-eqiad.php: Depool db1101:3317,3318
operations/mediawiki-config : masterdb-eqiad.php: Depool db1081
operations/mediawiki-config : masterdb-eqiad.php: Depool db1084
operations/mediawiki-config : masterdb-eqiad.php: Depool db1091
operations/mediawiki-config : masterdb-eqiad.php: Depool db1121
operations/mediawiki-config : masterdb-eqiad.php: Depool db1103:3314
operations/mediawiki-config : masterdb-eqiad.php: Depool db1097:3314
operations/mediawiki-config : masterdb-eqiad.php: Depool db1074
operations/mediawiki-config : masterdb-eqiad.php: Depool db1090:3312
operations/mediawiki-config : masterdb-eqiad.php: Depool db1076
operations/mediawiki-config : masterdb-eqiad.php: Fully depool db1105
operations/mediawiki-config : masterdb-eqiad.php: Depool db1122
operations/mediawiki-config : masterdb-eqiad.php: Depool db1103:3312
operations/mediawiki-config : masterdb-eqiad.php: Depool db1110
operations/mediawiki-config : masterdb-eqiad.php: Depool db1082
operations/mediawiki-config : masterdb-eqiad.php: Depool db1113:3315
operations/mediawiki-config : masterdb-eqiad.php: Depool db1100
operations/mediawiki-config : masterdb-eqiad.php: Depool db1097:3315
operations/mediawiki-config : masterdb-eqiad.php: Depool db1096:3315
operations/mediawiki-config : masterdb-eqiad.php: Depool db1088
operations/mediawiki-config : masterdb-eqiad.php: Depool db1093
operations/mediawiki-config : masterdb-eqiad.php: Depool db1085
operations/mediawiki-config : masterdb-eqiad.php: Depool db1113:3316
operations/mediawiki-config : masterdb-eqiad.php: Depool db1098:3316
operations/mediawiki-config : masterdb-eqiad.php: Depool db1096:3316
operations/mediawiki-config : masterdb-codfw.php: Depool db2060

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 490535 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/490535

Change 490535 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/490535

Mentioned in SAL (#wikimedia-operations) [2019-02-14T06:10:00Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1087 T210713 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2019-02-14T06:10:16Z] <marostegui> Deploy schema change on db1087 with replication, lag will be generated on labsdb:s8 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-14T07:57:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1087 T210713 (duration: 00m 54s)

Change 490620 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/490620

Change 490620 merged by Marostegui:
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/490620

Mentioned in SAL (#wikimedia-operations) [2019-02-14T16:39:02Z] <marostegui> Depool labsdb1009 - T210713

Change 490801 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/490801

Change 490801 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/490801

Mentioned in SAL (#wikimedia-operations) [2019-02-15T06:17:48Z] <marostegui> Deploy schema change on db1109 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T05:55:22Z] <marostegui> Deploy schema change on s8 primary master (db1071) - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T08:23:45Z] <marostegui> Deploy schema change on s1 codfw master (db2048), lag will be generated on s1 codfw - T210713

Marostegui updated the task description. (Show Details)Feb 18 2019, 9:02 AM
Marostegui added a comment.EditedFeb 18 2019, 9:04 AM

s1 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • dbstore1002
  • dbstore1001
  • db1124
  • db1119
  • db1118
  • db1106
  • db1105
  • db1099
  • db1089
  • db1083
  • db1080
  • db1067 T210713#4967984
Marostegui updated the task description. (Show Details)Feb 18 2019, 9:05 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:08:13Z] <marostegui> Deploy schema change on dbstore1003:3311 and dbstore1001:3311 - T210713

Change 491218 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/491218

Change 491218 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/491218

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:16:39Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T210713 (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:16:43Z] <marostegui> Deploy schema change on db1099:3311 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:46:47Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T210713 (duration: 00m 46s)

Change 491225 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/491225

Change 491225 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/491225

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:50:39Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:50:45Z] <marostegui> Deploy schema change on db1105:3311 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T10:54:10Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T210713 (duration: 00m 46s)

Change 491409 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/491409

Change 491409 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/491409

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:03Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1106 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:21Z] <marostegui> Deploy schema change on db1106 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:45Z] <marostegui> Deploy schema change on db1106 - this will generate lag on labsdb:s1 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:05:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1106 T210713 (duration: 00m 49s)

Change 491416 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/491416

Change 491416 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/491416

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:50:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1089 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:50:29Z] <marostegui> Deploy schema change on db1089 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T09:29:35Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1089 T210713 (duration: 00m 45s)

Change 491433 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/491433

Change 491433 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/491433

Mentioned in SAL (#wikimedia-operations) [2019-02-19T10:25:34Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1083 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T10:25:43Z] <marostegui> Deploy schema change on db1083 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:12:24Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1083 T210713 (duration: 00m 46s)

Change 491449 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1118

https://gerrit.wikimedia.org/r/491449

Change 491449 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1118

https://gerrit.wikimedia.org/r/491449

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:59:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1118 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:59:55Z] <marostegui> Deploy schema change on db1118 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T13:25:45Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1118 T210713 (duration: 00m 46s)

Change 491682 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/491682

Change 491682 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/491682

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:12:20Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1119 T210713 (duration: 01m 05s)

Change 491689 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/491689

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:44:32Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1119 T210713 (duration: 00m 51s)

Change 491689 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/491689

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:47:46Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 T210713 (duration: 00m 51s)

Mentioned in SAL (#wikimedia-operations) [2019-02-20T07:13:51Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1080 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-20T07:14:05Z] <marostegui> Deploy schema change on s1 primary master (db1067) - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-20T09:33:28Z] <marostegui> Deploy schema change on db2043 (s3 codfw master), lag will be generated on s3 codfw - T210713

Marostegui updated the task description. (Show Details)Feb 20 2019, 9:55 AM

db1067 (s1 master) has too much concurrency to let the alter go thru, I will try a few more times before giving up on it and leaving it for when we either failover the master or the DC.

Mentioned in SAL (#wikimedia-operations) [2019-02-20T10:04:06Z] <marostegui> Deploy schema change on dbstore1004:3313 - T210713

Marostegui added a comment.EditedFeb 20 2019, 10:04 AM

s3 eqiad

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • dbstore1002
  • db1124
  • db1123
  • db1095
  • db1077
  • db1075
  • db1078
Marostegui updated the task description. (Show Details)Feb 20 2019, 10:05 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-20T10:36:56Z] <marostegui> Deploy schema change on db1095:3313 - T210713

Change 491900 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/491900

Change 491900 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/491900

Mentioned in SAL (#wikimedia-operations) [2019-02-21T06:21:44Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1123 T210713 (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T06:22:08Z] <marostegui> Deploy schema change on db1123 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:08:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1123 T210713 (duration: 00m 55s)

Change 491906 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/491906

Change 491906 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/491906

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:12:54Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1077 T210713 (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:12:57Z] <marostegui> Deploy schema change on db1077 - this will generate lag on labsdb:s3 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:35:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1077 T210713 (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:40:36Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1075 T210713 (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:40:46Z] <marostegui> Deploy schema change on db1075 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T09:29:31Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1075 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T09:29:34Z] <marostegui> Deploy schema change on s3 primary master (db1078) - T210713

Change 491920 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] filtered_tables.txt: Remove change_tag.ct_tag column

https://gerrit.wikimedia.org/r/491920

Marostegui updated the task description. (Show Details)Feb 21 2019, 9:31 AM
Marostegui updated the task description. (Show Details)Feb 21 2019, 10:06 AM

All the hosts are done except db1067 (s1 master T210713#4967984 ) which I will try a few more times before stalling this until we do a failover.

Change 491920 merged by Marostegui:
[operations/puppet@production] filtered_tables.txt: Remove change_tag.ct_tag column

https://gerrit.wikimedia.org/r/491920

Marostegui updated the task description. (Show Details)Feb 21 2019, 10:24 AM
Marostegui changed the task status from Open to Stalled.Feb 26 2019, 8:02 AM

Stalling this until we have failed over s1 master, as it is impossible to alter that host whilst it is active.

Ladsgroup moved this task from Incoming to Monitor on the User-Ladsgroup board.Apr 17 2019, 7:02 PM
DannyS712 changed the task status from Stalled to Open.Thu, Nov 14, 6:41 AM
DannyS712 added a subscriber: DannyS712.

Stalling this until we have failed over s1 master, as it is impossible to alter that host whilst it is active.

Hehe, thanks! I am actually running the schema change already :)

Hehe, thanks! I am actually running the schema change already :)

db1067 has finally been altered:

root@db1067.eqiad.wmnet[enwiki]> ALTER TABLE /*_*/change_tag MODIFY ct_tag_id int unsigned NOT NULL;
Query OK, 92366667 rows affected (53 min 51.55 sec)
Records: 92366667  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]>
root@db1067.eqiad.wmnet[enwiki]> DROP INDEX /*i*/change_tag_rc_tag_nonuniq ON /*_*/change_tag;
Query OK, 0 rows affected (1.35 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]> DROP INDEX /*i*/change_tag_log_tag_nonuniq ON /*_*/change_tag;
Query OK, 0 rows affected (2.65 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]> DROP INDEX /*i*/change_tag_rev_tag_nonuniq ON /*_*/change_tag;
Query OK, 0 rows affected (0.48 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]> DROP INDEX /*i*/change_tag_tag_id ON /*_*/change_tag;
Query OK, 0 rows affected (2.65 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@db1067.eqiad.wmnet[enwiki]>
root@db1067.eqiad.wmnet[enwiki]> ALTER TABLE /*_*/change_tag DROP COLUMN ct_tag;
Query OK, 0 rows affected (34 min 49.92 sec)
Records: 0  Duplicates: 0  Warnings: 0
Marostegui closed this task as Resolved.Thu, Nov 14, 8:02 AM
Marostegui updated the task description. (Show Details)