Page MenuHomePhabricator

Drop change_tag.ct_tag column in production
Open, Stalled, NormalPublic

Description

The day has finally come:

  1. ALTERs to run: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/473601/6/maintenance/archives/patch-drop-ct_tag.sql
  2. Where to run those changes: all.dblist
  3. When to run those changes: After deployment of wmf.8, It would be great if we apply it in small and medium wikis first
  4. If the schema change is backwards compatible: No.
  5. If the schema change has been tested already on some of the test/beta wikis: Tested in beta cluster
  6. if the data should be made available on the labs replicas and/or dumps: Yes, data in this table is public (unless the wiki is private which we don't replicate at all)

Execution progress:

s1

s2

s3

s4

s5

s6

s7

s8

wikitech

  • eqiad
  • codfw

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 489968 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3318

https://gerrit.wikimedia.org/r/489968

Mentioned in SAL (#wikimedia-operations) [2019-02-12T06:14:42Z] <marostegui> Deploy schema change on db1099:3318 T210713

Marostegui updated the task description. (Show Details)Feb 12 2019, 8:07 AM

Change 489991 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/489991

Change 489991 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/489991

Mentioned in SAL (#wikimedia-operations) [2019-02-12T08:20:37Z] <marostegui> Depool db1101:3318 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-12T08:20:50Z] <marostegui> Deploy schema change on db1101:3318 - T210713

Change 490010 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/490010

Change 490010 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/490010

Mentioned in SAL (#wikimedia-operations) [2019-02-12T10:25:10Z] <marostegui> Deploy schema change on db1092 T210713

Change 490275 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/490275

Change 490275 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/490275

Mentioned in SAL (#wikimedia-operations) [2019-02-13T06:18:21Z] <marostegui> Deploy schema change on db1104 - T210713

Change 490535 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/490535

Change 490535 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/490535

Mentioned in SAL (#wikimedia-operations) [2019-02-14T06:10:00Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1087 T210713 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2019-02-14T06:10:16Z] <marostegui> Deploy schema change on db1087 with replication, lag will be generated on labsdb:s8 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-14T07:57:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1087 T210713 (duration: 00m 54s)

Change 490620 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/490620

Change 490620 merged by Marostegui:
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/490620

Mentioned in SAL (#wikimedia-operations) [2019-02-14T16:39:02Z] <marostegui> Depool labsdb1009 - T210713

Change 490801 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/490801

Change 490801 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/490801

Mentioned in SAL (#wikimedia-operations) [2019-02-15T06:17:48Z] <marostegui> Deploy schema change on db1109 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T05:55:22Z] <marostegui> Deploy schema change on s8 primary master (db1071) - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T08:23:45Z] <marostegui> Deploy schema change on s1 codfw master (db2048), lag will be generated on s1 codfw - T210713

Marostegui updated the task description. (Show Details)Feb 18 2019, 9:02 AM
Marostegui added a comment.EditedFeb 18 2019, 9:04 AM

s1 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • dbstore1002
  • dbstore1001
  • db1124
  • db1119
  • db1118
  • db1106
  • db1105
  • db1099
  • db1089
  • db1083
  • db1080
  • db1067 T210713#4967984
Marostegui updated the task description. (Show Details)Feb 18 2019, 9:05 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:08:13Z] <marostegui> Deploy schema change on dbstore1003:3311 and dbstore1001:3311 - T210713

Change 491218 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/491218

Change 491218 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/491218

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:16:39Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T210713 (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:16:43Z] <marostegui> Deploy schema change on db1099:3311 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:46:47Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T210713 (duration: 00m 46s)

Change 491225 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/491225

Change 491225 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/491225

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:50:39Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:50:45Z] <marostegui> Deploy schema change on db1105:3311 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T10:54:10Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T210713 (duration: 00m 46s)

Change 491409 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/491409

Change 491409 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/491409

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:03Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1106 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:21Z] <marostegui> Deploy schema change on db1106 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:45Z] <marostegui> Deploy schema change on db1106 - this will generate lag on labsdb:s1 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:05:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1106 T210713 (duration: 00m 49s)

Change 491416 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/491416

Change 491416 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/491416

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:50:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1089 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:50:29Z] <marostegui> Deploy schema change on db1089 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T09:29:35Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1089 T210713 (duration: 00m 45s)

Change 491433 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/491433

Change 491433 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/491433

Mentioned in SAL (#wikimedia-operations) [2019-02-19T10:25:34Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1083 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T10:25:43Z] <marostegui> Deploy schema change on db1083 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:12:24Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1083 T210713 (duration: 00m 46s)

Change 491449 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1118

https://gerrit.wikimedia.org/r/491449

Change 491449 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1118

https://gerrit.wikimedia.org/r/491449

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:59:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1118 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:59:55Z] <marostegui> Deploy schema change on db1118 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T13:25:45Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1118 T210713 (duration: 00m 46s)

Change 491682 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/491682

Change 491682 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/491682

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:12:20Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1119 T210713 (duration: 01m 05s)

Change 491689 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/491689

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:44:32Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1119 T210713 (duration: 00m 51s)

Change 491689 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/491689

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:47:46Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 T210713 (duration: 00m 51s)

Mentioned in SAL (#wikimedia-operations) [2019-02-20T07:13:51Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1080 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-20T07:14:05Z] <marostegui> Deploy schema change on s1 primary master (db1067) - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-20T09:33:28Z] <marostegui> Deploy schema change on db2043 (s3 codfw master), lag will be generated on s3 codfw - T210713

Marostegui updated the task description. (Show Details)Feb 20 2019, 9:55 AM

db1067 (s1 master) has too much concurrency to let the alter go thru, I will try a few more times before giving up on it and leaving it for when we either failover the master or the DC.

Mentioned in SAL (#wikimedia-operations) [2019-02-20T10:04:06Z] <marostegui> Deploy schema change on dbstore1004:3313 - T210713

Marostegui added a comment.EditedFeb 20 2019, 10:04 AM

s3 eqiad

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • dbstore1002
  • db1124
  • db1123
  • db1095
  • db1077
  • db1075
  • db1078
Marostegui updated the task description. (Show Details)Feb 20 2019, 10:05 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-20T10:36:56Z] <marostegui> Deploy schema change on db1095:3313 - T210713

Change 491900 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/491900

Change 491900 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/491900

Mentioned in SAL (#wikimedia-operations) [2019-02-21T06:21:44Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1123 T210713 (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T06:22:08Z] <marostegui> Deploy schema change on db1123 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:08:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1123 T210713 (duration: 00m 55s)

Change 491906 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/491906

Change 491906 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/491906

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:12:54Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1077 T210713 (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:12:57Z] <marostegui> Deploy schema change on db1077 - this will generate lag on labsdb:s3 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:35:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1077 T210713 (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:40:36Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1075 T210713 (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:40:46Z] <marostegui> Deploy schema change on db1075 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T09:29:31Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1075 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T09:29:34Z] <marostegui> Deploy schema change on s3 primary master (db1078) - T210713

Change 491920 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] filtered_tables.txt: Remove change_tag.ct_tag column

https://gerrit.wikimedia.org/r/491920

Marostegui updated the task description. (Show Details)Feb 21 2019, 9:31 AM
Marostegui updated the task description. (Show Details)Feb 21 2019, 10:06 AM

All the hosts are done except db1067 (s1 master T210713#4967984 ) which I will try a few more times before stalling this until we do a failover.

Change 491920 merged by Marostegui:
[operations/puppet@production] filtered_tables.txt: Remove change_tag.ct_tag column

https://gerrit.wikimedia.org/r/491920

Marostegui updated the task description. (Show Details)Feb 21 2019, 10:24 AM
Marostegui changed the task status from Open to Stalled.Tue, Feb 26, 8:02 AM

Stalling this until we have failed over s1 master, as it is impossible to alter that host whilst it is active.