Page MenuHomePhabricator
Create Task
Maniphest T233135

Schema change for refactored actor and comment storage
Closed, ResolvedPublic
Actions

Description

From the checklist at https://wikitech.wikimedia.org/wiki/Schema_changes:

  1. ALTERs to run: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/482371/3/maintenance/archives/patch-drop-comment-fields.sql and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/525605/8/maintenance/archives/patch-drop-user-fields.sql
  2. Where to run those changes: all.dblist
  3. When to run those changes: No time constraint.
  4. If the schema change is backwards compatible: Yes. The code no longer accesses these fields.
  5. If the schema change has been tested already on some of the test/beta wikis: It appears to have been auto-deployed to Beta. Things seem to be working, and if anyone has complained about things breaking I haven't heard it.
  6. If the data should be made available on the labs replicas and/or dumps: N/A, no new data is being added here. References to these fields from Labs replica views were removed in T212972 and T223406.

Progress:

  • s1
    • codfw
    • eqiad progress T233135#5691987
    • Run maintain-views on labs hosts to recreate the views (sudo /usr/local/sbin/maintain-views --databases jawiki --table protected_titles --replace-all)
  • s2
    • codfw
    • eqiad progress T233135#5578395
    • Run maintain-views on labs hosts to recreate the views
  • s3
    • codfw
    • eqiad progress T233135#5703954
    • Run maintain-views on labs hosts to recreate the views
  • s4
    • codfw
    • eqiad progress T233135#5730827
    • Run maintain-views on labs hosts to recreate the views
  • s5
    • codfw
    • eqiad progress T233135#5655648
    • Run maintain-views on labs hosts to recreate the views
  • s7
    • codfw
    • eqiad progress T233135#5571998
    • Run maintain-views on labs hosts to recreate the views
  • s8
    • codfw
    • eqiad progress T233135#5671879
    • Run maintain-views on labs hosts to recreate the views
  • labswiki
  • labtestwiki

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
filtered_tables.txt: Remove dropped columnsoperations/puppetproduction+0 -18
index-conf.yaml: Remove unused indexoperations/puppetproduction+1 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui added a comment.EditedOct 16 2019, 5:16 AM

s2 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • db1129
  • db1125
  • db1122
  • db1105
  • db1103
  • db1095
  • db1090
  • db1076
  • db1074
Stashbot added a comment.Oct 16 2019, 5:18 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-16T05:18:28Z] <marostegui> Deploy schema change on s2 sanitarium master (db1074) this will create lag on s2 labsdb T233135 T234066

Marostegui updated the task description. (Show Details)Oct 16 2019, 5:56 AM
Stashbot added a comment.Oct 16 2019, 10:17 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-16T10:17:10Z] <marostegui> Stop replication on s2 codfw master for schema change and to modify sanitarium triggers T234066 T233135 T234704

Stashbot mentioned this in T234704: Remove ar_comment from sanitarium triggers.Oct 16 2019, 10:17 AM
Marostegui closed subtask T233625: Change PK and remove partitions from the logging table as Resolved.Oct 16 2019, 12:19 PM
Stashbot added a comment.Oct 16 2019, 3:09 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-16T15:09:17Z] <marostegui> Recreate views for protected_titles on s2 and s7 on labsdb1009 and labsdb1012 - T233135

Stashbot added a comment.Oct 16 2019, 3:17 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-16T15:17:27Z] <marostegui> Deploy schema change on dbstore1004:3312 - T234066 T233135

Marostegui updated the task description. (Show Details)Oct 17 2019, 5:04 AM
Marostegui updated the task description. (Show Details)Oct 17 2019, 5:25 AM
Marostegui updated the task description. (Show Details)Oct 17 2019, 5:30 AM
Stashbot added a comment.Oct 18 2019, 5:15 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-18T05:15:54Z] <marostegui> Deploy schema change on db1129 T233135 T234066

Stashbot added a comment.Oct 21 2019, 5:09 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-21T05:09:38Z] <marostegui> Deploy schema change on s7 primary master db1062 - T234066 T233135

Stashbot added a comment.Oct 21 2019, 5:14 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-21T05:14:28Z] <marostegui> Deploy schema change on db1090:3312 T234066 T233135

Marostegui updated the task description. (Show Details)Oct 21 2019, 9:25 AM
Stashbot added a comment.Oct 21 2019, 1:04 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-21T13:04:47Z] <marostegui> Deploy schema change on db1122 (s2 primary master) - T233135 T234066

• Nuria added a project: Analytics.EditedOct 21 2019, 7:50 PM
• Nuria added subscribers: JAllemandou, Milimetric, • Nuria.

Pinging analytics temporarily so we know these changes are happening, it should not affect scooping of these tables cause i think we generate schema every time we scoop but want to make sure. cc @Milimetric @JAllemandou In terms of algorithms we have not used these fields for a while.

Marostegui updated the task description. (Show Details)Oct 22 2019, 4:58 AM
JAllemandou added a comment.Oct 22 2019, 8:00 PM
In T233135#5592951, @Nuria wrote:

Pinging analytics temporarily so we know these changes are happening, it shoudl not affect scooping of these tables cause i think we generate schema everytime we scoop but want to make sure. cc @Milimetric @JAllemandou In terms of algorithms we have not used these fields for a while.

I confirm the fields are not used in our current sqoop queries. The change should be transparent for us.

• Nuria removed a project: Analytics.Oct 22 2019, 8:10 PM
• Nuria unsubscribed.
Ladsgroup subscribed.Oct 23 2019, 8:50 PM
jcrespo mentioned this in Unknown Object (Task).Nov 6 2019, 10:10 AM
jcrespo mentioned this in T237509: maintain views has to run or has to be updated to fix errors on globalblocks and protected_titles for wikireplicas.Nov 6 2019, 10:39 AM
Stashbot added a comment.Nov 12 2019, 6:40 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T06:40:51Z] <marostegui> Deploy schema change on s5 codfw with replication, this will generate lag on s5 codfw T233135 T234066

Marostegui updated the task description. (Show Details)Nov 12 2019, 10:32 AM
Marostegui added a comment.EditedNov 12 2019, 11:35 AM

s5 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • db1130
  • db1124
  • db1113
  • db1110
  • db1102
  • db1100
  • db1097
  • db1096
  • db1082
Stashbot added a comment.Nov 12 2019, 3:41 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T15:41:29Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3315 for a schema change T233135 T234066', diff saved to https://phabricator.wikimedia.org/P9600 and previous config saved to /var/cache/conftool/dbconfig/20191112-154127-marostegui.json

Stashbot added a comment.Nov 12 2019, 3:49 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T15:49:26Z] <marostegui> Deploy schema change on db1102:3315 T233135 T234066

Marostegui updated the task description. (Show Details)Nov 13 2019, 8:06 AM
Marostegui updated the task description. (Show Details)
Marostegui mentioned this in T231711: Unable to access SpecialNewpages on enwiki: WMFTimeoutException.Nov 14 2019, 7:07 AM
Stashbot added a comment.Nov 14 2019, 8:41 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-14T08:41:30Z] <marostegui> Deploy schema change with replication on db1082, this will generate lag on s5 labs - T233135 T234066

Stashbot added a comment.Nov 15 2019, 5:57 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-15T05:57:28Z] <marostegui> Run maintain-views for s5 on labsdb1009, labsdb1010, labsdb1012 (pending labsdb1011 as it is still running the schema change) T233135

Stashbot added a comment.Nov 15 2019, 10:45 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-15T10:45:27Z] <marostegui> Run maintain-views for s5 on labsdb1011 T233135

Marostegui updated the task description. (Show Details)Nov 15 2019, 10:46 AM
Stashbot added a comment.Nov 18 2019, 5:53 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-18T05:53:57Z] <marostegui> Deploy schema change on s5 primary master db1100 - T233135 T234066

Marostegui updated the task description. (Show Details)Nov 18 2019, 9:09 AM
Stashbot added a comment.Nov 18 2019, 9:11 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-18T09:11:12Z] <marostegui> Deploy schema change on s8 codfw, this will generate lag on s8 codfw - T233135 T234066

Marostegui updated the task description. (Show Details)Nov 18 2019, 2:18 PM
Marostegui added a comment.EditedNov 18 2019, 2:23 PM

s8 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1005
  • db1126
  • db1124
  • db1116
  • db1109
  • db1104
  • db1101
  • db1099
  • db1092
  • db1087
Marostegui updated the task description. (Show Details)Nov 19 2019, 5:28 AM
Stashbot added a comment.Nov 25 2019, 2:37 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-25T14:37:26Z] <marostegui> Deploy schema change on s1 codfw (this will generate lag on codfw) - T234066 T233135

• WDoranWMF mentioned this in T238966: Apply updates for MCR, actor migration, and content migration, to production wikis..Nov 25 2019, 4:22 PM
Marostegui updated the task description. (Show Details)Nov 26 2019, 5:47 AM
Marostegui added a comment.EditedNov 26 2019, 5:50 AM

s1 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • db1139
  • db1134
  • db1124
  • db1119
  • db1118
  • db1114
  • db1106
  • db1105
  • db1099
  • db1089
  • db1083
  • db1080
Stashbot added a comment.Nov 26 2019, 9:18 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-26T09:18:36Z] <marostegui> Run maintain-views for wikidatawiki.protected_title view on labsdb hosts T233135

Stashbot added a comment.Nov 26 2019, 9:26 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-26T09:26:50Z] <marostegui> Deploy schema change on s8 primary master (db1109) - T234066 T233135 T237120

Stashbot mentioned this in T237120: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language.Nov 26 2019, 9:26 AM
Marostegui updated the task description. (Show Details)Nov 26 2019, 11:33 AM
Marostegui updated the task description. (Show Details)Nov 27 2019, 7:44 AM
Stashbot added a comment.Nov 28 2019, 1:37 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-28T13:37:03Z] <marostegui> Recreate views for enwiki_p.protected_titles for all labsdb hosts - T233135

Stashbot added a comment.Nov 28 2019, 1:37 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-28T13:37:26Z] <marostegui> Deploy schema change on db1106 with replication (lag will appear on s1 on labs) - T234066 T233135

Marostegui updated the task description. (Show Details)Nov 28 2019, 1:40 PM
Marostegui updated the task description. (Show Details)Nov 29 2019, 9:20 AM
Marostegui updated the task description. (Show Details)Nov 29 2019, 6:25 PM
Marostegui added a comment.EditedDec 2 2019, 5:55 AM

s3 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • db1124
  • db1123
  • db1112
  • db1095
  • db1078
  • db1075
Marostegui updated the task description. (Show Details)Dec 2 2019, 6:25 AM
Marostegui updated the task description. (Show Details)Dec 4 2019, 5:57 AM
Marostegui updated the task description. (Show Details)Dec 5 2019, 5:43 AM
Stashbot added a comment.Dec 10 2019, 6:08 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-10T06:08:50Z] <marostegui> Deploy schema change on s4 codfw master (this will generate lag on s4 codfw) T233135

Marostegui updated the task description. (Show Details)Dec 11 2019, 5:43 AM
Marostegui added a comment.EditedDec 11 2019, 6:30 AM

s4 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • db1138 T233135#5748152
  • db1125
  • db1121
  • db1103
  • db1102
  • db1097
  • db1091
  • db1084
  • db1081
Marostegui updated the task description. (Show Details)Dec 11 2019, 6:31 AM
Stashbot added a comment.Dec 11 2019, 10:45 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-11T10:45:08Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1103:3314 for schema change T233135', diff saved to https://phabricator.wikimedia.org/P9851 and previous config saved to /var/cache/conftool/dbconfig/20191211-104506-marostegui.json

Stashbot added a comment.Dec 11 2019, 7:51 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-11T19:51:32Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1103:3314 after schema change T233135', diff saved to https://phabricator.wikimedia.org/P9857 and previous config saved to /var/cache/conftool/dbconfig/20191211-195130-marostegui.json

Stashbot added a comment.Dec 11 2019, 7:53 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-11T19:53:08Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3314 for schema change T233135', diff saved to https://phabricator.wikimedia.org/P9858 and previous config saved to /var/cache/conftool/dbconfig/20191211-195306-marostegui.json

Stashbot added a comment.Dec 12 2019, 5:47 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-12T05:47:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1097:3314 after schema change T233135', diff saved to https://phabricator.wikimedia.org/P9861 and previous config saved to /var/cache/conftool/dbconfig/20191212-054708-marostegui.json

Stashbot added a comment.Dec 16 2019, 6:39 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-16T06:39:43Z] <marostegui> Recreate views on commonswiki,testcommonswiki for protected_titles on all labsdb hosts - T233135

Marostegui updated the task description. (Show Details)Dec 16 2019, 6:40 AM
Marostegui closed subtask T234704: Remove ar_comment from sanitarium triggers as Resolved.Dec 16 2019, 8:53 AM
Marostegui added a comment.EditedDec 17 2019, 4:24 PM

s4 is almost done, only pending the master (db1138), which I am alter slowly and table by table manually as those tables are pretty big and very accessed, progress on the master:

  • archive
  • ipblocks
  • oldimage
  • filearchive
  • recentchanges
  • protected_titles
  • image
  • logging
Stashbot added a comment.Dec 17 2019, 4:49 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-17T16:49:49Z] <marostegui> Deploy schema change on commonswiki.logging on db1138 (s4 primary master) - T233135

Stashbot added a comment.Dec 18 2019, 5:31 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-18T05:31:35Z] <marostegui> Deploy schema change on commonswiki.image on s4 primary master (db1138) - T233135

Marostegui updated the task description. (Show Details)Dec 18 2019, 6:59 AM
gerritbot added a comment.Dec 18 2019, 7:05 AM

Change 558851 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] filtered_tables.txt: Remove dropped columns

https://gerrit.wikimedia.org/r/558851

gerritbot added a project: Patch-For-Review.Dec 18 2019, 7:05 AM
Marostegui added a comment.Dec 18 2019, 7:05 AM

All the alters have been done, just pending a review and merge: https://gerrit.wikimedia.org/r/558851

Marostegui updated the task description. (Show Details)Dec 18 2019, 7:05 AM
Marostegui moved this task from In progress to Done on the Schema-change-in-production board.Dec 18 2019, 7:10 AM
Marostegui added a comment.Dec 18 2019, 4:41 PM

@Anomie if you can review https://gerrit.wikimedia.org/r/558851 that's what's left to be able to close this task \o/

gerritbot added a comment.Dec 19 2019, 6:10 AM

Change 558851 merged by Marostegui:
[operations/puppet@production] filtered_tables.txt: Remove dropped columns

https://gerrit.wikimedia.org/r/558851

Marostegui closed this task as Resolved.Dec 19 2019, 6:13 AM
Marostegui updated the task description. (Show Details)

All done

Maintenance_bot removed a project: Patch-For-Review.Dec 19 2019, 7:10 AM
Andru mentioned this in T257356: 1.33.X update to shared ipblocks table fails after first run.Jul 8 2020, 8:47 AM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:41 PM
Marostegui mentioned this in T158166: Discuss labsdb visibility of rev_text_id and ar_comment.Jul 2 2024, 7:53 AM
Maintenance_bot moved this task from In progress to Done on the DBA board.Jul 2 2024, 8:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits