Page MenuHomePhabricator

Schema change for refactored actor storage
Closed, ResolvedPublic

Description

From the checklist at https://wikitech.wikimedia.org/wiki/Schema_changes:

  1. ALTERs to run: https://gerrit.wikimedia.org/r/#/c/380669/48/maintenance/archives/patch-actor-table.sql, minus the changes to the image table that are already being done per T187089#3966227
  2. Where to run those changes: all.dblist
  3. When to run those changes: No time constraint.
  4. If the schema change is backwards compatible: Yes. The new columns and tables won't be used until a feature flag is enabled.
  5. If the schema change has been tested already on some of the test/beta wikis: It has been auto-deployed to Beta. I haven't heard that it broke anything.
  6. if the data should be made available on the labs replicas and/or dumps: Treatment of replicas and dumps are the same.
  • All columns in the actor table can be made available without restriction. Rows must only be available if (1) actor_user refers to a visible row in the user table or (2) the row is publicly referenced from one of the _actor columns being added in this patch. Extensions converted to have _actor columns will eventually need to be added to that list. Note: From the MediaWiki side, this table should receive manly INSERTs, no UPDATEs, and DELETEs only due to potential maintenance. Also note {T91706}.
  • All columns in revision_actor_temp can be made available without restriction. Rows must only be available if the corresponding revision table row exists (join on revactor_rev = rev_id) and has (rev_deleted & 4) = 0. Note: From the MediaWiki side, this table should receive INSERTs and DELETEs but no UPDATEs.
  • All the _actor columns added to existing tables should be available under the same conditions as the corresponding _user or _user_text columns. These seem to be:
    • archive.ar_actor: Available when (ar_deleted & 4) = 0.
    • ipblocks.ipb_by_actor: Always available (whenever the row itself is).
    • image.img_actor: Always available.
    • oldimage.oi_actor: Available when (oi_deleted & 4) = 0.
    • filearchive.fa_actor: Available when (fa_deleted & 4) = 0.
    • recentchanges.rc_actor: Available when (rc_deleted & 4) = 0.
    • logging.log_actor: Available when (log_deleted & 4) = 0.

Schema change progress:

Details

Related Gerrit Patches:
operations/mediawiki-config : masterdb-eqiad.php: Depool db1083
operations/mediawiki-config : masterdb-eqiad.php: Depool db1106
operations/mediawiki-config : masterdb-eqiad.php: Depool db1119
operations/mediawiki-config : masterdb-eqiad.php: Depool db1105:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1119
operations/mediawiki-config : masterdb-eqiad.php: Depool db1114
operations/mediawiki-config : masterdb-eqiad.php: Depool db1089
operations/mediawiki-config : masterdb-eqiad.php: Depool db1099:3311
operations/mediawiki-config : masterdb-eqiad.php: Depool db1067
operations/mediawiki-config : masterdb-eqiad.php: Depool db1080
operations/mediawiki-config : masterdb-eqiad.php: Depool db1109
operations/mediawiki-config : masterdb-eqiad.php: Depool db1104
operations/mediawiki-config : masterdb-eqiad.php: Depool db1087
operations/mediawiki-config : masterdb-eqiad.php: Depool db1101:3318
operations/mediawiki-config : masterdb-eqiad.php: Depool db1099:3318
operations/mediawiki-config : masterdb-eqiad.php: Depool db1104
operations/mediawiki-config : masterdb-eqiad.php: Depool db1092
operations/mediawiki-config : masterdb-eqiad.php: Depool db1078
operations/mediawiki-config : masterdb-eqiad.php: Depool db1077
operations/mediawiki-config : masterdb-eqiad.php: Depool db1123
operations/mediawiki-config : masterdb-eqiad.php: Depool db1090:3317
operations/mediawiki-config : masterdb-eqiad.php: Depool db1079
operations/mediawiki-config : masterdb-eqiad.php: Depool db1098:3317
operations/mediawiki-config : masterdb-eqiad.php: Depool db1101:3317
operations/mediawiki-config : masterdb-eqiad.php: Depool db1094
operations/mediawiki-config : masterdb-eqiad.php: Depool db1086
operations/mediawiki-config : masterdb-eqiad.php: Depool db1091
operations/mediawiki-config : masterdb-eqiad.php: Depool db1103:3314
operations/mediawiki-config : masterdb-eqiad.php: Depool db1121
operations/mediawiki-config : masterdb-eqiad.php: Depool db1097:3314
operations/mediawiki-config : masterdb-eqiad.php: Depool db1084
operations/mediawiki-config : masterdb-eqiad.php: Depool db1081
operations/mediawiki-config : masterdb-eqiad.php: Depool db1076
operations/mediawiki-config : masterdb-eqiad.php: Depool db1074
operations/mediawiki-config : masterdb-eqiad.php: Depool db1105:3312
operations/mediawiki-config : masterdb-eqiad.php: Depool db1103:3312
operations/mediawiki-config : masterdb-eqiad.php: Depool db1085
operations/mediawiki-config : masterdb-eqiad.php: Depool db1113:3316
operations/mediawiki-config : masterdb-eqiad.php: Depool db1098:3316
operations/mediawiki-config : masterdb-eqiad.php: Depool db1088
operations/mediawiki-config : masterdb-eqiad.php: Depool db1093
operations/mediawiki-config : masterdb-eqiad.php: Depool db1096:3316
operations/mediawiki-config : masterdb-eqiad.php: Depool db1110
operations/mediawiki-config : masterdb-eqiad.php: Depool db1082
operations/mediawiki-config : masterdb-eqiad.php: Depool db1113:3315
operations/mediawiki-config : masterdb-eqiad.php: Depool db1097:3315
operations/mediawiki-config : masterdb-eqiad.php: Depool db1100
operations/mediawiki-config : masterdb-eqiad.php: Depool db1096:3315

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2018-05-17T07:22:28Z] <marostegui> Deploy schema change on db1090:3317 - T191519 T188299 T190148

Mentioned in SAL (#wikimedia-operations) [2018-05-17T12:50:30Z] <marostegui> Deploy schema change on s3 codfw primary master (db2043) this will generate lag on codfw - T191519 T188299 T190148

Marostegui updated the task description. (Show Details)May 17 2018, 2:27 PM
Marostegui updated the task description. (Show Details)EditedMay 18 2018, 5:28 AM

s3 eqiad progress

  • labsdb1009
  • labsdb1010
  • labsdb1011
  • dbstore1002
  • db1095
  • db1116
  • db1077
  • db1078
  • db1123
  • db1075
Marostegui updated the task description. (Show Details)May 18 2018, 5:31 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-18T05:33:25Z] <marostegui> Deploy schema change on dbstore1002:s3 - T191519 T188299 T190148

Mentioned in SAL (#wikimedia-operations) [2018-05-21T05:09:39Z] <marostegui> Deploy schema change on s7 primary master (db1062) - T191519 T188299 T1901482

Change 434298 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/434298

Change 434298 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1123

https://gerrit.wikimedia.org/r/434298

Mentioned in SAL (#wikimedia-operations) [2018-05-21T05:40:56Z] <marostegui> Deploy schema change on db1123 - T191519 T188299 T1901482

Marostegui updated the task description. (Show Details)May 21 2018, 7:47 AM

Change 434302 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/434302

Change 434302 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1077

https://gerrit.wikimedia.org/r/434302

Mentioned in SAL (#wikimedia-operations) [2018-05-21T08:02:38Z] <marostegui> Deploy schema change on db1077 with replication, this will generate lags on labs - T191519 T188299 T1901482

Change 434434 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1078

https://gerrit.wikimedia.org/r/434434

Change 434434 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1078

https://gerrit.wikimedia.org/r/434434

Mentioned in SAL (#wikimedia-operations) [2018-05-22T05:10:33Z] <marostegui> Deploy schema change on db1078 - T191519 T188299 T1901482

Mentioned in SAL (#wikimedia-operations) [2018-05-22T07:21:26Z] <marostegui> Deploy schema change on s8 codfw master (db2045) with replication, this will generate lags on codfw - T191519 T188299 T190148

Marostegui updated the task description. (Show Details)May 22 2018, 8:30 AM
Marostegui added a comment.EditedMay 22 2018, 12:18 PM

s8 eqiad progress:

  • dbstore1002
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • db1095
  • db1109
  • db1104
  • db1116
  • db1101
  • db1099
  • db1092
  • db1087
  • db1071

Mentioned in SAL (#wikimedia-operations) [2018-05-23T05:06:26Z] <marostegui> Deploy schema change on s3 primary master (db1075) - T191519 T188299 T190148

Mentioned in SAL (#wikimedia-operations) [2018-05-23T05:21:48Z] <marostegui> Deploy schema change on dbstore1002:s8 - T191519 T188299 T190148 T194273 T194270

Change 434624 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/434624

Change 434624 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1092

https://gerrit.wikimedia.org/r/434624

Mentioned in SAL (#wikimedia-operations) [2018-05-23T05:43:48Z] <marostegui> Deploy schema change on db1092 - T191519 T188299 T190148 T194273 T194270

Change 434631 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/434631

Change 434631 abandoned by Marostegui:
db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/434631

Marostegui updated the task description. (Show Details)May 23 2018, 7:17 AM
Marostegui updated the task description. (Show Details)May 23 2018, 8:11 AM

Change 434640 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3318

https://gerrit.wikimedia.org/r/434640

Change 434640 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3318

https://gerrit.wikimedia.org/r/434640

Mentioned in SAL (#wikimedia-operations) [2018-05-23T08:21:11Z] <marostegui> Deploy schema change on db1099:3318 - T191519 T188299 T190148 T194270

Change 434648 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/434648

Change 434648 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318

https://gerrit.wikimedia.org/r/434648

Mentioned in SAL (#wikimedia-operations) [2018-05-23T09:30:36Z] <marostegui> Deploy schema change on db1101:3318 - T191519 T188299 T190148 T194270

Change 434868 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/434868

Change 434868 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1087

https://gerrit.wikimedia.org/r/434868

Mentioned in SAL (#wikimedia-operations) [2018-05-24T06:17:40Z] <marostegui> Deploy schema change on db1087, this will generate lag on labs on s8 - T191519 T188299 T190148 T194270

Change 434872 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/434872

Change 434872 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/434872

Mentioned in SAL (#wikimedia-operations) [2018-05-24T06:47:02Z] <marostegui> Deploy schema change on db1104 - T191519 T188299 T190148 T194270

Change 434875 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/434875

Change 434875 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/434875

Mentioned in SAL (#wikimedia-operations) [2018-05-24T07:36:27Z] <marostegui> Deploy schema change on db1109 - T191519 T188299 T190148 T194270

Mentioned in SAL (#wikimedia-operations) [2018-05-24T08:05:15Z] <marostegui> Deploy schema change on s8 primary master (db1071) - T191519 T188299 T190148 T194270

Marostegui updated the task description. (Show Details)May 24 2018, 8:42 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-24T08:45:55Z] <marostegui> Deploy schema change on s1 codfw primary master (db2048), this will generate lag on codfw - T191519 T188299 T190148

Mentioned in SAL (#wikimedia-operations) [2018-05-24T11:10:12Z] <marostegui> Deploy schema change on dbstore1002:s1 - T191519 T188299 T190148

Marostegui updated the task description. (Show Details)EditedMay 24 2018, 11:11 AM

s1 eqiad progress

  • labsdb1009
  • labsdb1010
  • labsdb1011
  • db1095
  • dbstore1001
  • dbstore1002
  • db1116
  • db1080
  • db1083
  • db1089
  • db1119
  • db1067
  • db1106
  • db1099
  • db1114
  • db1105
  • db1052

Change 434899 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/434899

Change 434899 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1080

https://gerrit.wikimedia.org/r/434899

Mentioned in SAL (#wikimedia-operations) [2018-05-24T11:19:07Z] <marostegui> Deploy schema change on db1080 - T191519 T188299 T190148

Marostegui updated the task description. (Show Details)May 24 2018, 1:21 PM

Change 434911 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1067

https://gerrit.wikimedia.org/r/434911

Change 434911 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1067

https://gerrit.wikimedia.org/r/434911

Mentioned in SAL (#wikimedia-operations) [2018-05-24T13:33:15Z] <marostegui> Deploy schema change on db1067 - T191519 T188299 T190148

Mentioned in SAL (#wikimedia-operations) [2018-05-24T14:57:39Z] <marostegui> Deploy schema change on dbstore1001:s1 - T191519 T188299 T190148

Change 434944 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/434944

Change 434944 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099:3311

https://gerrit.wikimedia.org/r/434944

Mentioned in SAL (#wikimedia-operations) [2018-05-24T16:19:15Z] <marostegui> Deploy schema change on db1099:3311 - T191519 T188299 T190148

Change 435112 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/435112

Change 435112 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1089

https://gerrit.wikimedia.org/r/435112

Mentioned in SAL (#wikimedia-operations) [2018-05-25T05:23:15Z] <marostegui> Deploy schema change on db1089 - T190148 T191519 T188299

Change 435120 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1114

https://gerrit.wikimedia.org/r/435120

Change 435120 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1114

https://gerrit.wikimedia.org/r/435120

Mentioned in SAL (#wikimedia-operations) [2018-05-25T06:57:14Z] <marostegui> Deploy schema change on db1114 - T190148 T191519 T188299

Change 435135 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/435135

Change 435135 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/435135

Change 435137 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/435137

Change 435137 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1105:3311

https://gerrit.wikimedia.org/r/435137

Mentioned in SAL (#wikimedia-operations) [2018-05-25T09:20:29Z] <marostegui> Deploy schema change on db1105:3311 - T190148 T191519 T188299

Change 435166 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/435166

Change 435166 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1119

https://gerrit.wikimedia.org/r/435166

Change 435723 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/435723

Change 435723 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1106

https://gerrit.wikimedia.org/r/435723

Mentioned in SAL (#wikimedia-operations) [2018-05-28T05:23:36Z] <marostegui> Deploy schema change on db1106 with replication, this will generate lag on labs - T190148 T191519 T188299

Change 435729 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/435729

Change 435729 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1083

https://gerrit.wikimedia.org/r/435729

Mentioned in SAL (#wikimedia-operations) [2018-05-28T07:31:32Z] <marostegui> Deploy schema change on db1083 - T190148 T191519 T188299

Marostegui removed a project: Data-Services.

Only s1 master is now pending, which will be done probably on Wednesday EU morning.
I have created a different task for the labs views handling (T195747) as this task is a bit messy now with all the comments, commits etc.

Mentioned in SAL (#wikimedia-operations) [2018-05-30T05:06:13Z] <marostegui> Deploy schema change on s1 primary master db1052 - T188299

Marostegui closed this task as Resolved.May 30 2018, 9:11 AM
Marostegui updated the task description. (Show Details)

All done.
Note there is still this ticket pending in regards to the actor storage refactoring: T187089

Thanks for reminding me about that, I might have forgotten that the image table was being done in that task rather than this one.

Are we at the point that I should deploy the change here? https://gerrit.wikimedia.org/r/#/c/431823 --presuming I fix the complaints from the jenkins-bot.

Probably not, since img_actor doesn't exist yet.

Change 431823 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] wiki replicas - prepare for refactored actor storage

https://gerrit.wikimedia.org/r/431823