Page MenuHomePhabricator

Schema change for refactored actor and comment storage
Closed, ResolvedPublic

Description

From the checklist at https://wikitech.wikimedia.org/wiki/Schema_changes:

  1. ALTERs to run: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/482371/3/maintenance/archives/patch-drop-comment-fields.sql and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/525605/8/maintenance/archives/patch-drop-user-fields.sql
  2. Where to run those changes: all.dblist
  3. When to run those changes: No time constraint.
  4. If the schema change is backwards compatible: Yes. The code no longer accesses these fields.
  5. If the schema change has been tested already on some of the test/beta wikis: It appears to have been auto-deployed to Beta. Things seem to be working, and if anyone has complained about things breaking I haven't heard it.
  6. If the data should be made available on the labs replicas and/or dumps: N/A, no new data is being added here. References to these fields from Labs replica views were removed in T212972 and T223406.

Progress:

  • s1
    • codfw
    • eqiad progress T233135#5691987
    • Run maintain-views on labs hosts to recreate the views (sudo /usr/local/sbin/maintain-views --databases jawiki --table protected_titles --replace-all)
  • s2
    • codfw
    • eqiad progress T233135#5578395
    • Run maintain-views on labs hosts to recreate the views
  • s3
    • codfw
    • eqiad progress T233135#5703954
    • Run maintain-views on labs hosts to recreate the views
  • s4
    • codfw
    • eqiad progress T233135#5730827
    • Run maintain-views on labs hosts to recreate the views
  • s5
    • codfw
    • eqiad progress T233135#5655648
    • Run maintain-views on labs hosts to recreate the views
  • s7
    • codfw
    • eqiad progress T233135#5571998
    • Run maintain-views on labs hosts to recreate the views
  • s8
    • codfw
    • eqiad progress T233135#5671879
    • Run maintain-views on labs hosts to recreate the views
  • labswiki
  • labtestwiki

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2019-10-15T05:28:49Z] <marostegui> Deploy schema change on db1098:3317 T234066 T233135

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:08:05Z] <marostegui> Deploy schema change on s7 sanitarium master (db1079) this will create lag on s7 labsdb T233135 T234066

Marostegui added a comment.EditedOct 16 2019, 5:16 AM

s2 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • db1129
  • db1125
  • db1122
  • db1105
  • db1103
  • db1095
  • db1090
  • db1076
  • db1074

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:18:28Z] <marostegui> Deploy schema change on s2 sanitarium master (db1074) this will create lag on s2 labsdb T233135 T234066

Mentioned in SAL (#wikimedia-operations) [2019-10-16T10:17:10Z] <marostegui> Stop replication on s2 codfw master for schema change and to modify sanitarium triggers T234066 T233135 T234704

Mentioned in SAL (#wikimedia-operations) [2019-10-16T15:09:17Z] <marostegui> Recreate views for protected_titles on s2 and s7 on labsdb1009 and labsdb1012 - T233135

Mentioned in SAL (#wikimedia-operations) [2019-10-16T15:17:27Z] <marostegui> Deploy schema change on dbstore1004:3312 - T234066 T233135

Mentioned in SAL (#wikimedia-operations) [2019-10-18T05:15:54Z] <marostegui> Deploy schema change on db1129 T233135 T234066

Mentioned in SAL (#wikimedia-operations) [2019-10-21T05:09:38Z] <marostegui> Deploy schema change on s7 primary master db1062 - T234066 T233135

Mentioned in SAL (#wikimedia-operations) [2019-10-21T05:14:28Z] <marostegui> Deploy schema change on db1090:3312 T234066 T233135

Mentioned in SAL (#wikimedia-operations) [2019-10-21T13:04:47Z] <marostegui> Deploy schema change on db1122 (s2 primary master) - T233135 T234066

Nuria added a project: Analytics.EditedOct 21 2019, 7:50 PM
Nuria added subscribers: JAllemandou, Milimetric, Nuria.

Pinging analytics temporarily so we know these changes are happening, it should not affect scooping of these tables cause i think we generate schema every time we scoop but want to make sure. cc @Milimetric @JAllemandou In terms of algorithms we have not used these fields for a while.

Pinging analytics temporarily so we know these changes are happening, it shoudl not affect scooping of these tables cause i think we generate schema everytime we scoop but want to make sure. cc @Milimetric @JAllemandou In terms of algorithms we have not used these fields for a while.

I confirm the fields are not used in our current sqoop queries. The change should be transparent for us.

Nuria removed a subscriber: Nuria.
jcrespo mentioned this in Unknown Object (Task).Nov 6 2019, 10:10 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T06:40:51Z] <marostegui> Deploy schema change on s5 codfw with replication, this will generate lag on s5 codfw T233135 T234066

Marostegui added a comment.EditedNov 12 2019, 11:35 AM

s5 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • db1130
  • db1124
  • db1113
  • db1110
  • db1102
  • db1100
  • db1097
  • db1096
  • db1082

Mentioned in SAL (#wikimedia-operations) [2019-11-12T15:41:29Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3315 for a schema change T233135 T234066', diff saved to https://phabricator.wikimedia.org/P9600 and previous config saved to /var/cache/conftool/dbconfig/20191112-154127-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-11-12T15:49:26Z] <marostegui> Deploy schema change on db1102:3315 T233135 T234066

Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-11-14T08:41:30Z] <marostegui> Deploy schema change with replication on db1082, this will generate lag on s5 labs - T233135 T234066

Mentioned in SAL (#wikimedia-operations) [2019-11-15T05:57:28Z] <marostegui> Run maintain-views for s5 on labsdb1009, labsdb1010, labsdb1012 (pending labsdb1011 as it is still running the schema change) T233135

Mentioned in SAL (#wikimedia-operations) [2019-11-15T10:45:27Z] <marostegui> Run maintain-views for s5 on labsdb1011 T233135

Mentioned in SAL (#wikimedia-operations) [2019-11-18T05:53:57Z] <marostegui> Deploy schema change on s5 primary master db1100 - T233135 T234066

Mentioned in SAL (#wikimedia-operations) [2019-11-18T09:11:12Z] <marostegui> Deploy schema change on s8 codfw, this will generate lag on s8 codfw - T233135 T234066

Marostegui added a comment.EditedNov 18 2019, 2:23 PM

s8 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1005
  • db1126
  • db1124
  • db1116
  • db1109
  • db1104
  • db1101
  • db1099
  • db1092
  • db1087

Mentioned in SAL (#wikimedia-operations) [2019-11-25T14:37:26Z] <marostegui> Deploy schema change on s1 codfw (this will generate lag on codfw) - T234066 T233135

Marostegui added a comment.EditedNov 26 2019, 5:50 AM

s1 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • db1139
  • db1134
  • db1124
  • db1119
  • db1118
  • db1114
  • db1106
  • db1105
  • db1099
  • db1089
  • db1083
  • db1080

Mentioned in SAL (#wikimedia-operations) [2019-11-26T09:18:36Z] <marostegui> Run maintain-views for wikidatawiki.protected_title view on labsdb hosts T233135

Mentioned in SAL (#wikimedia-operations) [2019-11-26T09:26:50Z] <marostegui> Deploy schema change on s8 primary master (db1109) - T234066 T233135 T237120

Mentioned in SAL (#wikimedia-operations) [2019-11-28T13:37:03Z] <marostegui> Recreate views for enwiki_p.protected_titles for all labsdb hosts - T233135

Mentioned in SAL (#wikimedia-operations) [2019-11-28T13:37:26Z] <marostegui> Deploy schema change on db1106 with replication (lag will appear on s1 on labs) - T234066 T233135

Marostegui added a comment.EditedDec 2 2019, 5:55 AM

s3 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • db1124
  • db1123
  • db1112
  • db1095
  • db1078
  • db1075

Mentioned in SAL (#wikimedia-operations) [2019-12-10T06:08:50Z] <marostegui> Deploy schema change on s4 codfw master (this will generate lag on s4 codfw) T233135

Marostegui added a comment.EditedDec 11 2019, 6:30 AM

s4 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • db1138 T233135#5748152
  • db1125
  • db1121
  • db1103
  • db1102
  • db1097
  • db1091
  • db1084
  • db1081

Mentioned in SAL (#wikimedia-operations) [2019-12-11T10:45:08Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1103:3314 for schema change T233135', diff saved to https://phabricator.wikimedia.org/P9851 and previous config saved to /var/cache/conftool/dbconfig/20191211-104506-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-12-11T19:51:32Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1103:3314 after schema change T233135', diff saved to https://phabricator.wikimedia.org/P9857 and previous config saved to /var/cache/conftool/dbconfig/20191211-195130-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-12-11T19:53:08Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3314 for schema change T233135', diff saved to https://phabricator.wikimedia.org/P9858 and previous config saved to /var/cache/conftool/dbconfig/20191211-195306-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-12-12T05:47:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1097:3314 after schema change T233135', diff saved to https://phabricator.wikimedia.org/P9861 and previous config saved to /var/cache/conftool/dbconfig/20191212-054708-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-12-16T06:39:43Z] <marostegui> Recreate views on commonswiki,testcommonswiki for protected_titles on all labsdb hosts - T233135

Marostegui added a comment.EditedDec 17 2019, 4:24 PM

s4 is almost done, only pending the master (db1138), which I am alter slowly and table by table manually as those tables are pretty big and very accessed, progress on the master:

  • archive
  • ipblocks
  • oldimage
  • filearchive
  • recentchanges
  • protected_titles
  • image
  • logging

Mentioned in SAL (#wikimedia-operations) [2019-12-17T16:49:49Z] <marostegui> Deploy schema change on commonswiki.logging on db1138 (s4 primary master) - T233135

Mentioned in SAL (#wikimedia-operations) [2019-12-18T05:31:35Z] <marostegui> Deploy schema change on commonswiki.image on s4 primary master (db1138) - T233135

Change 558851 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] filtered_tables.txt: Remove dropped columns

https://gerrit.wikimedia.org/r/558851

All the alters have been done, just pending a review and merge: https://gerrit.wikimedia.org/r/558851

@Anomie if you can review https://gerrit.wikimedia.org/r/558851 that's what's left to be able to close this task \o/

Change 558851 merged by Marostegui:
[operations/puppet@production] filtered_tables.txt: Remove dropped columns

https://gerrit.wikimedia.org/r/558851

Marostegui closed this task as Resolved.Dec 19 2019, 6:13 AM
Marostegui updated the task description. (Show Details)

All done

Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:41 PM