Page MenuHomePhabricator

Drop change_tag.ct_tag column in production
Open, Stalled, NormalPublic

Description

The day has finally come:

  1. ALTERs to run: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/473601/6/maintenance/archives/patch-drop-ct_tag.sql
  2. Where to run those changes: all.dblist
  3. When to run those changes: After deployment of wmf.8, It would be great if we apply it in small and medium wikis first
  4. If the schema change is backwards compatible: No.
  5. If the schema change has been tested already on some of the test/beta wikis: Tested in beta cluster
  6. if the data should be made available on the labs replicas and/or dumps: Yes, data in this table is public (unless the wiki is private which we don't replicate at all)

Execution progress:

s1

s2

s3

s4

s5

s6

s7

s8

wikitech

  • eqiad
  • codfw

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2019-02-12T06:14:42Z] <marostegui> Deploy schema change on db1099:3318 T210713

Marostegui updated the task description. (Show Details)Feb 12 2019, 8:07 AM

Change 489991 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/489991

Change 489991 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/489991

Mentioned in SAL (#wikimedia-operations) [2019-02-12T08:20:37Z] <marostegui> Depool db1101:3318 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-12T08:20:50Z] <marostegui> Deploy schema change on db1101:3318 - T210713

Change 490010 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/490010

Change 490010 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/490010

Mentioned in SAL (#wikimedia-operations) [2019-02-12T10:25:10Z] <marostegui> Deploy schema change on db1092 T210713

Change 490275 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/490275

Change 490275 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/490275

Mentioned in SAL (#wikimedia-operations) [2019-02-13T06:18:21Z] <marostegui> Deploy schema change on db1104 - T210713

Change 490535 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/490535

Change 490535 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/490535

Mentioned in SAL (#wikimedia-operations) [2019-02-14T06:10:00Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1087 T210713 (duration: 00m 55s)

Mentioned in SAL (#wikimedia-operations) [2019-02-14T06:10:16Z] <marostegui> Deploy schema change on db1087 with replication, lag will be generated on labsdb:s8 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-14T07:57:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1087 T210713 (duration: 00m 54s)

Change 490620 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/490620

Change 490620 merged by Marostegui:
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/490620

Mentioned in SAL (#wikimedia-operations) [2019-02-14T16:39:02Z] <marostegui> Depool labsdb1009 - T210713

Change 490801 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/490801

Change 490801 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/490801

Mentioned in SAL (#wikimedia-operations) [2019-02-15T06:17:48Z] <marostegui> Deploy schema change on db1109 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T05:55:22Z] <marostegui> Deploy schema change on s8 primary master (db1071) - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T08:23:45Z] <marostegui> Deploy schema change on s1 codfw master (db2048), lag will be generated on s1 codfw - T210713

Marostegui updated the task description. (Show Details)Feb 18 2019, 9:02 AM
Marostegui added a comment.EditedFeb 18 2019, 9:04 AM

s1 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • dbstore1002
  • dbstore1001
  • db1124
  • db1119
  • db1118
  • db1106
  • db1105
  • db1099
  • db1089
  • db1083
  • db1080
  • db1067 T210713#4967984
Marostegui updated the task description. (Show Details)Feb 18 2019, 9:05 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:08:13Z] <marostegui> Deploy schema change on dbstore1003:3311 and dbstore1001:3311 - T210713

Change 491218 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/491218

Change 491218 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/491218

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:16:39Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T210713 (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:16:43Z] <marostegui> Deploy schema change on db1099:3311 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:46:47Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T210713 (duration: 00m 46s)

Change 491225 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/491225

Change 491225 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/491225

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:50:39Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-18T09:50:45Z] <marostegui> Deploy schema change on db1105:3311 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-18T10:54:10Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T210713 (duration: 00m 46s)

Change 491409 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/491409

Change 491409 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/491409

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:03Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1106 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:21Z] <marostegui> Deploy schema change on db1106 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T06:56:45Z] <marostegui> Deploy schema change on db1106 - this will generate lag on labsdb:s1 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:05:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1106 T210713 (duration: 00m 49s)

Change 491416 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/491416

Change 491416 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/491416

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:50:06Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1089 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T08:50:29Z] <marostegui> Deploy schema change on db1089 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T09:29:35Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1089 T210713 (duration: 00m 45s)

Change 491433 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/491433

Change 491433 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/491433

Mentioned in SAL (#wikimedia-operations) [2019-02-19T10:25:34Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1083 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T10:25:43Z] <marostegui> Deploy schema change on db1083 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:12:24Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1083 T210713 (duration: 00m 46s)

Change 491449 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1118

https://gerrit.wikimedia.org/r/491449

Change 491449 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1118

https://gerrit.wikimedia.org/r/491449

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:59:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1118 T210713 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-02-19T11:59:55Z] <marostegui> Deploy schema change on db1118 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-19T13:25:45Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1118 T210713 (duration: 00m 46s)

Change 491682 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/491682

Change 491682 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/491682

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:12:20Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1119 T210713 (duration: 01m 05s)

Change 491689 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/491689

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:44:32Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1119 T210713 (duration: 00m 51s)

Change 491689 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/491689

Mentioned in SAL (#wikimedia-operations) [2019-02-20T06:47:46Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 T210713 (duration: 00m 51s)

Mentioned in SAL (#wikimedia-operations) [2019-02-20T07:13:51Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1080 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-20T07:14:05Z] <marostegui> Deploy schema change on s1 primary master (db1067) - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-20T09:33:28Z] <marostegui> Deploy schema change on db2043 (s3 codfw master), lag will be generated on s3 codfw - T210713

Marostegui updated the task description. (Show Details)Feb 20 2019, 9:55 AM

db1067 (s1 master) has too much concurrency to let the alter go thru, I will try a few more times before giving up on it and leaving it for when we either failover the master or the DC.

Mentioned in SAL (#wikimedia-operations) [2019-02-20T10:04:06Z] <marostegui> Deploy schema change on dbstore1004:3313 - T210713

Marostegui added a comment.EditedFeb 20 2019, 10:04 AM

s3 eqiad

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • dbstore1002
  • db1124
  • db1123
  • db1095
  • db1077
  • db1075
  • db1078
Marostegui updated the task description. (Show Details)Feb 20 2019, 10:05 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-20T10:36:56Z] <marostegui> Deploy schema change on db1095:3313 - T210713

Change 491900 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/491900

Change 491900 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/491900

Mentioned in SAL (#wikimedia-operations) [2019-02-21T06:21:44Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1123 T210713 (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T06:22:08Z] <marostegui> Deploy schema change on db1123 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:08:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1123 T210713 (duration: 00m 55s)

Change 491906 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/491906

Change 491906 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/491906

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:12:54Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1077 T210713 (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T07:12:57Z] <marostegui> Deploy schema change on db1077 - this will generate lag on labsdb:s3 T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:35:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1077 T210713 (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:40:36Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1075 T210713 (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T08:40:46Z] <marostegui> Deploy schema change on db1075 - T210713

Mentioned in SAL (#wikimedia-operations) [2019-02-21T09:29:31Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1075 T210713 (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2019-02-21T09:29:34Z] <marostegui> Deploy schema change on s3 primary master (db1078) - T210713

Change 491920 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] filtered_tables.txt: Remove change_tag.ct_tag column

https://gerrit.wikimedia.org/r/491920

Marostegui updated the task description. (Show Details)Feb 21 2019, 9:31 AM
Marostegui updated the task description. (Show Details)Feb 21 2019, 10:06 AM

All the hosts are done except db1067 (s1 master T210713#4967984 ) which I will try a few more times before stalling this until we do a failover.

Change 491920 merged by Marostegui:
[operations/puppet@production] filtered_tables.txt: Remove change_tag.ct_tag column

https://gerrit.wikimedia.org/r/491920

Marostegui updated the task description. (Show Details)Feb 21 2019, 10:24 AM
Marostegui changed the task status from Open to Stalled.Feb 26 2019, 8:02 AM

Stalling this until we have failed over s1 master, as it is impossible to alter that host whilst it is active.

Ladsgroup moved this task from Incoming to Monitor on the User-Ladsgroup board.Apr 17 2019, 7:02 PM