Page MenuHomePhabricator

Dropping user.user_options on wmf databases
Closed, ResolvedPublic

Description

DBA Update

The change is prepared at https://docs.google.com/document/d/1mw4CQxYyNP2PruzlG30dPVQjVOqJTC3Q_DLViIhWrm0/edit?usp=sharing

The alter to run is:

ALTER TABLE user DROP COLUMN IF EXISTS user_options;

Where to run the alter: s1, s2, s3, s4, s5, s6, s7, s8
When to run: start at 2018-11-14
If the schema change is backwards compatible: compatible with the current code deployed
If the schema change has been tested already on some of the test/beta wikis.: wikis created since 2011-09-28 doesn't have this column
If it is involves new columns to tables: no, it's a drop

Phases

Testing

  • db2046
  • db1088

Execution

  • s1
  • s2
  • s3
  • s4
  • s5
  • s6
  • s7
  • s8

s6

  • db2039 codfw master, execute with replication!
  • db1088
  • db1093
  • db1113
  • db1098
  • db1096
  • dbstore1002
  • dbstore1001
  • db1085 Sanitarium master, execute with replication!
  • db1061 MASTER

s2

  • db2035 codfw master, execute with replication!
  • db1076
  • db1090
  • db1095
  • db1103
  • db1105
  • db1122
  • dbstore1002
  • db1074 Sanitarium master , execute with replication!
  • db1066 MASTER

s5

  • db2052 codfw master, execute with replication!
  • db1096
  • db1097
  • db1100
  • db1102
  • db1110
  • db1113
  • dbstore1002
  • db1082 Sanitarium master , execute with replication!
  • db1070 MASTER

s4

  • db2051 codfw master, execute with replication!
  • db1081
  • db1084
  • db1091
  • db1097
  • db1102
  • db1103
  • dbstore1002
  • db1121 Sanitarium master , execute with replication!
  • db1068 MASTER

s7

  • db2040 codfw master, execute with replication!
  • db1086
  • db1090
  • db1094
  • db1098
  • db1101
  • db1116
  • dbstore1002
  • db1079 Sanitarium master , execute with replication!
  • db1062 MASTER

s8

  • db2045 codfw master, execute with replication!
  • db1092
  • db1099
  • db1101
  • db1104
  • db1109
  • db1116
  • db1087 Sanitarium master , execute with replication!
  • db1071 MASTER

s1

  • db2048 codfw master, execute with replication!
  • db1080
  • db1083
  • db1089
  • db1099
  • db1105
  • db1114
  • db1119
  • dbstore1002
  • dbstore1001
  • db1106 Sanitarium master , execute with replication!
  • db1067 MASTER

s3

  • db2043 codfw master, execute with replication!
  • db1075
  • db1095
  • db1123
  • dbstore1002
  • db1077 Sanitarium master , execute with replication!
  • db1078 MASTER

In CODFW (Secondary datacenter) the alter table command we need to use is:

./wmfmariadbpy/wmfmariadbpy/osc_host.py --method=ddl --host <section_codfw_master>.codfw.wmnet --dblist mediawiki-config/dblists/<section>.dblist --table user "DROP COLUMN IF EXISTS user_options"

On EQIAD (primary datacenter) the alter command will be (on all hots one by one, depooled except sanitarium master where we need replication):

./wmfmariadbpy/wmfmariadbpy/osc_host.py --no-repicate --method=ddl --host <hostname>.codfw.wmnet --dblist mediawiki-config/dblists/<section>.dblist --table user "DROP COLUMN IF EXISTS user_options"

On Sanitarium hosts we need to recreate the TRIGGERS as:

/home/banyek/T85757/fix_triggers.sh db2094:3315 s5

Original text
The column user.user_options is visible on ToolLabs (at least on dewiki_p). I assume that the field also exists on wmf wikis.

It was dropped with http://git.wikimedia.org/commit/mediawiki%2Fcore.git/eda06e8593c12b4359a46cf3b428c1a1a88e40c4 and should be dropped in production as well.

If it already dropped from production wikis, please move this task to ToolLabs to get the field also dropped there. Thanks.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2019-01-07T10:30:49Z] <banyek> repooling db1079 after schema change - T85757

Mentioned in SAL (#wikimedia-operations) [2019-01-07T10:36:31Z] <banyek@deploy1001> Synchronized wmf-config/db-eqiad.php: repool db1079 after schema change - T85757 (duration: 00m 44s)

Banyek updated the task description. (Show Details)Jan 7 2019, 10:40 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-08T10:25:44Z] <banyek> executing schema change on db1062 - T85757

on db1062 (s7 master) every database is done, except eswiki I have to retry this later. (Lock wait timeout exceeded)

Banyek updated the task description. (Show Details)Jan 10 2019, 10:42 AM

on db1062 (s7 master) every database is done, except eswiki I have to retry this later. (Lock wait timeout exceeded)

finally I was able to run the alter there too

on db1062 (s7 master) every database is done, except eswiki I have to retry this later. (Lock wait timeout exceeded)

finally I was able to run the alter there too

Awesome news! I will take it over now and get the pending sections done.
Thanks for getting most of it done!

Thank you <3

Mentioned in SAL (#wikimedia-operations) [2019-01-10T18:37:55Z] <marostegui> Stop replication on s8 codfw master for a schema change - T85757

s8 (wikidatawiki) doesn't have the column. As that wiki is relatively "new" I guess it was created after the patch was originally merged.

Marostegui updated the task description. (Show Details)Jan 10 2019, 6:44 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-10T18:46:03Z] <marostegui> Stop replication on s1 codfw master for a schema change - T85757

Mentioned in SAL (#wikimedia-operations) [2019-01-10T18:47:41Z] <marostegui> Deploy schema change on s1 codfw master (db2048) with replication, this will generate lag on s1 codfw - T85757

Marostegui updated the task description. (Show Details)Jan 10 2019, 7:12 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-10T19:13:53Z] <marostegui> Deploy schema change on dbstore1002 - T85757

Marostegui updated the task description. (Show Details)Jan 10 2019, 7:14 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-10T19:14:12Z] <marostegui> Deploy schema change on dbstore1001 - T85757

The change is prepared at https://docs.google.com/document/d/1mw4CQxYyNP2PruzlG30dPVQjVOqJTC3Q_DLViIhWrm0/edit?usp=sharing

@Banyek It would be very great that this task description contained the change itself, not a link to a not WMF-hosted platform, and that it not publicly available!

Marostegui updated the task description. (Show Details)Jan 10 2019, 7:19 PM
Marostegui updated the task description. (Show Details)Jan 10 2019, 11:06 PM

Change 484241 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/484241

Change 484241 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/484241

Mentioned in SAL (#wikimedia-operations) [2019-01-14T15:16:51Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db1105:3311 T85757 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-01-14T15:16:55Z] <marostegui> Deploy schema change on db1105:3311 - T85757

Marostegui updated the task description. (Show Details)Jan 14 2019, 3:58 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-14T16:02:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db1105:3311 T85757 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-01-14T16:03:50Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T85757 (duration: 00m 45s)

Change 484356 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/484356

Change 484356 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/484356

Mentioned in SAL (#wikimedia-operations) [2019-01-15T07:07:18Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T85757 (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2019-01-15T07:07:21Z] <marostegui> Deploy schema change on db1099:3311 - T85757

Marostegui updated the task description. (Show Details)Jan 15 2019, 7:51 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-15T07:56:09Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T85757 (duration: 00m 46s)

Change 484360 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] Bug: T85757

https://gerrit.wikimedia.org/r/484360

Mentioned in SAL (#wikimedia-operations) [2019-01-15T08:02:16Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1089 T85757 (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2019-01-15T08:02:58Z] <marostegui> Deploy schema change on db1089 - T85757

Marostegui updated the task description. (Show Details)Jan 15 2019, 8:20 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-15T08:23:28Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1089 T85757 (duration: 00m 46s)

Change 484365 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/484365

Change 484365 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/484365

Mentioned in SAL (#wikimedia-operations) [2019-01-15T08:28:25Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1106 T85757 (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2019-01-15T08:28:29Z] <marostegui> Deploy schema change on db1106 - T85757

Mentioned in SAL (#wikimedia-operations) [2019-01-15T08:38:21Z] <marostegui> Stop replication on s1 on all labs hosts - T85757

Marostegui updated the task description. (Show Details)Jan 15 2019, 9:10 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-15T09:45:26Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1106 T85757 (duration: 00m 46s)

Change 484382 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1114

https://gerrit.wikimedia.org/r/484382

Change 484382 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1114

https://gerrit.wikimedia.org/r/484382

Mentioned in SAL (#wikimedia-operations) [2019-01-15T09:51:00Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1114 T85757 (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2019-01-15T09:51:34Z] <marostegui> Deploy schema change on db1114 - T85757

Marostegui updated the task description. (Show Details)Jan 15 2019, 10:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-15T10:13:51Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1114 T85757 (duration: 00m 45s)

Change 484391 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/484391

Change 484391 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/484391

Mentioned in SAL (#wikimedia-operations) [2019-01-15T10:20:05Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 T85757 (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2019-01-15T10:20:50Z] <marostegui> Deploy schema change on db1080 - T85757

Marostegui updated the task description. (Show Details)Jan 15 2019, 10:35 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-15T10:40:44Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1080 T85757 (duration: 00m 45s)

Change 484402 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/484402

Change 484402 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/484402

Mentioned in SAL (#wikimedia-operations) [2019-01-15T10:56:51Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1083 T85757 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-01-15T10:57:00Z] <marostegui> Deploy schema change on db1083 - T85757

Marostegui updated the task description. (Show Details)Jan 15 2019, 11:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-15T11:10:30Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1083 T85757 (duration: 00m 45s)

Change 484418 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/484418

Change 484418 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/484418

Mentioned in SAL (#wikimedia-operations) [2019-01-15T13:15:21Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1119 T85757 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-01-15T13:15:29Z] <marostegui> Deploy schema change on db1119 - T85757

Marostegui updated the task description. (Show Details)Jan 15 2019, 1:31 PM

Mentioned in SAL (#wikimedia-operations) [2019-01-15T13:33:52Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1119 T85757 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-01-16T06:06:46Z] <marostegui> Deploy schema change on db1067 (s1 primary master) - T85757

Mentioned in SAL (#wikimedia-operations) [2019-01-17T08:31:35Z] <marostegui> Deploy schema change on s3 codfw, lag will be generated - T85757

Marostegui updated the task description. (Show Details)Jan 17 2019, 9:12 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-17T09:18:51Z] <marostegui> Deploy schema change on db1095:3313 - T85757

Marostegui updated the task description. (Show Details)Jan 17 2019, 10:17 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-18T06:29:39Z] <marostegui> Deploy schema change on db1075 - T85757

Marostegui updated the task description. (Show Details)Jan 18 2019, 6:44 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-18T06:46:21Z] <marostegui> Deploy schema change on dbstore1002:s3 - T85757

Marostegui updated the task description. (Show Details)Jan 18 2019, 6:53 AM

Change 485580 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/485580

Change 485580 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/485580

Mentioned in SAL (#wikimedia-operations) [2019-01-21T06:54:30Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1123 - T85757 (duration: 00m 50s)

Mentioned in SAL (#wikimedia-operations) [2019-01-21T06:54:35Z] <marostegui> Deploy schema change on db1123 - T85757

Mentioned in SAL (#wikimedia-operations) [2019-01-21T07:39:39Z] <marostegui> Stop replication on db1124:3313 to fix triggers - T85757

Marostegui updated the task description. (Show Details)Jan 21 2019, 8:20 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-21T08:24:03Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1123 - T85757 (duration: 00m 48s)

Change 485588 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/485588

Change 485588 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/485588

Mentioned in SAL (#wikimedia-operations) [2019-01-21T08:31:59Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1077 - T85757 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2019-01-21T08:35:51Z] <marostegui> Stop replication db1077 to deploy schema change - T85757

Marostegui updated the task description. (Show Details)Jan 21 2019, 9:50 AM

Mentioned in SAL (#wikimedia-operations) [2019-01-21T10:36:13Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1077 - T85757 (duration: 00m 44s)

Mentioned in SAL (#wikimedia-operations) [2019-01-22T06:42:56Z] <marostegui> Deploy schema change on db1078 (s3 master) - T85757

Marostegui updated the task description. (Show Details)Jan 22 2019, 7:47 AM
Marostegui updated the task description. (Show Details)Jan 22 2019, 8:08 AM
Marostegui updated the task description. (Show Details)
Marostegui closed this task as Resolved.

All done