Page MenuHomePhabricator
Create Task
Maniphest T276292

Schema change for renaming new_name_timestamp to rc_new_name_timestamp in recentchanges
Closed, ResolvedPublic
Actions

Description

Part of T270033: Fix and enforce table prefix usage in columns and indexes in core

  1. ALTERs to run: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/667358/4/maintenance/archives/patch-recentchanges-rc_new_name_timestamp.sql
  2. Where to run those changes: all.dblist
  3. When to run those changes: After 1.36-wmf.34 is deployed everywhere (Monday 15 March)
  4. If the schema change is backwards compatible: Yes
  5. If the schema change has been tested already on some of the test/beta wikis: Tested in beta cluster.
  6. if the data should be made available on the labs replicas and/or dumps: Yes, data in this table is public

Progress

Details

SubjectRepoBranchLines +/-
Use `rc_new_name_timestamp` index name unconditionallymediawiki/coremaster+1 -5
Customize query in gerrit

Related Objects

Event Timeline

Ladsgroup created this task.Mar 2 2021, 11:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2021, 11:50 PM
Marostegui triaged this task as Medium priority.Mar 3 2021, 5:35 AM
Marostegui added a project: DBA.
Marostegui moved this task from Triage to Ready on the DBA board.
Marostegui subscribed.Mar 18 2021, 9:57 AM

@Ladsgroup this can be deployed?
As always with indexes renames, I would deploy this change to a host in s6 eqiad, and leave it a few days running to see if we get query errors (some queries doing FORCE index or similar).

Ladsgroup added a comment.Mar 18 2021, 10:05 AM

Yeah it should be safe now. The fix for the change got deployed now (https://gerrit.wikimedia.org/g/mediawiki/core/+/7addd74ec3fbcd361ef330e1128e2da1463aa2b5/maintenance/categoryChangesAsRdf.php)

Marostegui added a comment.Mar 18 2021, 10:05 AM

Thank you!

Ladsgroup updated the task description. (Show Details)Mar 18 2021, 10:19 AM
Marostegui claimed this task.Apr 14 2021, 11:24 AM
Marostegui updated the task description. (Show Details)
Marostegui moved this task from Ready to In progress on the DBA board.Apr 14 2021, 11:27 AM
Marostegui moved this task from Backlog to In progress on the Schema-change-in-production board.
Marostegui added a comment.Apr 14 2021, 11:40 AM

@Ladsgroup I have altered db1096:3316 and will leave it running for a few days, to make sure we have no code forcing the old index:

# mysql.py -hdb1096:3316 frwiki -e "show create table recentchanges\G"
*************************** 1. row ***************************
       Table: recentchanges
Create Table: CREATE TABLE `recentchanges` (
  `rc_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `rc_timestamp` binary(14) NOT NULL,
  `rc_actor` bigint(20) unsigned NOT NULL,
  `rc_namespace` int(11) NOT NULL DEFAULT 0,
  `rc_title` varbinary(255) NOT NULL DEFAULT '',
  `rc_comment_id` bigint(20) unsigned NOT NULL,
  `rc_minor` tinyint(3) unsigned NOT NULL DEFAULT 0,
  `rc_bot` tinyint(3) unsigned NOT NULL DEFAULT 0,
  `rc_new` tinyint(3) unsigned NOT NULL DEFAULT 0,
  `rc_cur_id` int(10) unsigned NOT NULL DEFAULT 0,
  `rc_this_oldid` int(10) unsigned NOT NULL DEFAULT 0,
  `rc_last_oldid` int(10) unsigned NOT NULL DEFAULT 0,
  `rc_type` tinyint(3) unsigned NOT NULL DEFAULT 0,
  `rc_source` varbinary(16) NOT NULL DEFAULT '',
  `rc_patrolled` tinyint(3) unsigned NOT NULL DEFAULT 0,
  `rc_ip` varbinary(40) NOT NULL DEFAULT '',
  `rc_old_len` int(10) DEFAULT NULL,
  `rc_new_len` int(10) DEFAULT NULL,
  `rc_deleted` tinyint(1) unsigned NOT NULL DEFAULT 0,
  `rc_logid` int(10) unsigned NOT NULL DEFAULT 0,
  `rc_log_type` varbinary(255) DEFAULT NULL,
  `rc_log_action` varbinary(255) DEFAULT NULL,
  `rc_params` blob DEFAULT NULL,
  PRIMARY KEY (`rc_id`),
  KEY `rc_timestamp` (`rc_timestamp`),
  KEY `rc_cur_id` (`rc_cur_id`),
  KEY `rc_ip` (`rc_ip`),
  KEY `rc_name_type_patrolled_timestamp` (`rc_namespace`,`rc_type`,`rc_patrolled`,`rc_timestamp`),
  KEY `rc_ns_actor` (`rc_namespace`,`rc_actor`),
  KEY `rc_actor` (`rc_actor`,`rc_timestamp`),
  KEY `rc_namespace_title_timestamp` (`rc_namespace`,`rc_title`,`rc_timestamp`),
  KEY `rc_this_oldid` (`rc_this_oldid`),
  KEY `rc_new_name_timestamp` (`rc_new`,`rc_namespace`,`rc_timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=438991179 DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8

This is on: frwiki jawiki ruwiki

Marostegui added a comment.EditedApr 14 2021, 11:42 AM

s6 progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1005
  • db2141
  • db2129
  • db2124
  • db2117
  • db2114
  • db2097
  • db2095
  • db2089
  • db2087
  • db2076
  • db1180
  • db1173
  • db1168
  • db1165
  • db1155
  • db1140
  • db1139
  • db1131
  • db1125
  • db1113
  • db1098
  • db1096
  • db1085
  • clouddb1021
  • clouddb1019
  • clouddb1015
Marostegui updated the task description. (Show Details)Apr 14 2021, 11:43 AM
Marostegui added a comment.Apr 15 2021, 6:07 AM

Altered also db1098:3316

Marostegui added a comment.Apr 15 2021, 12:33 PM

I am not seeing any errors coming from any index force or related, I will wait till Monday though before proceeding

Marostegui updated the task description. (Show Details)Apr 19 2021, 7:15 AM
Marostegui updated the task description. (Show Details)Apr 19 2021, 7:58 AM
Marostegui updated the task description. (Show Details)Apr 21 2021, 6:24 AM
Marostegui added a comment.EditedApr 21 2021, 6:26 AM

s5 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • db1161
  • db1154
  • db1150
  • db1145
  • db1144
  • db1130
  • db1124
  • db1113
  • db1110
  • db1100
  • db1096
  • db1082
  • clouddb1021
  • clouddb1020
  • clouddb1016
Marostegui added a comment.Apr 21 2021, 6:45 AM

s5 fully done

Marostegui updated the task description. (Show Details)Apr 21 2021, 6:45 AM
Marostegui updated the task description. (Show Details)Apr 21 2021, 6:50 AM
Marostegui updated the task description. (Show Details)Apr 21 2021, 9:52 AM
Marostegui added a comment.EditedApr 21 2021, 9:54 AM

s2 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • db1182
  • db1171
  • db1170
  • db1162
  • db1156
  • db1155
  • db1146
  • db1129
  • db1125
  • db1122
  • db1105
  • db1074
  • clouddb1021
  • clouddb1018
  • clouddb1014
Marostegui updated the task description. (Show Details)Apr 21 2021, 9:55 AM
Marostegui updated the task description. (Show Details)Apr 23 2021, 5:35 AM
Marostegui updated the task description. (Show Details)Apr 23 2021, 6:24 AM
Marostegui added a comment.EditedApr 23 2021, 6:26 AM

s8 eqiad progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1005
  • db1177
  • db1172
  • db1154
  • db1126
  • db1116
  • db1114
  • db1111
  • db1109
  • db1104
  • db1101
  • db1099
  • db1087
  • clouddb1021
  • clouddb1020
  • clouddb1016
Marostegui updated the task description. (Show Details)Apr 23 2021, 6:27 AM
Marostegui updated the task description. (Show Details)Apr 23 2021, 1:01 PM
Marostegui updated the task description. (Show Details)Apr 26 2021, 5:29 AM
Marostegui added a comment.EditedApr 26 2021, 5:37 AM

s1 eqiad

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • db1184
  • db1169
  • db1164
  • db1163
  • db1154
  • db1140
  • db1139
  • db1135
  • db1134
  • db1133
  • db1119
  • db1118
  • db1106
  • db1105
  • db1099
  • db1083
  • clouddb1021
  • clouddb1017
  • clouddb1013
Marostegui added a comment.Apr 26 2021, 5:48 AM

s1 is just waiting for the master, which will be done once the switchover on Wednesday is done (T278214)

Marostegui updated the task description. (Show Details)Apr 26 2021, 5:48 AM
Stashbot added a comment.Apr 26 2021, 10:25 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-26T10:25:10Z] <marostegui> Deploy schema change on s4 codfw, lag will appear T276292

Marostegui updated the task description. (Show Details)Apr 26 2021, 10:25 AM
Marostegui added a comment.EditedApr 26 2021, 10:38 AM

s4 progress

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • db1160
  • db1155
  • db1150
  • db1149
  • db1148
  • db1147
  • db1146
  • db1145
  • db1144
  • db1143
  • db1142
  • db1141
  • db1138
  • db1121
  • clouddb1021
  • clouddb1019
  • clouddb1015
Marostegui updated the task description. (Show Details)Apr 26 2021, 11:42 AM
Marostegui updated the task description. (Show Details)Apr 27 2021, 5:27 AM
Marostegui added a comment.EditedApr 27 2021, 5:32 AM

s7 eqiad

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1003
  • db1181
  • db1174
  • db1170
  • db1158
  • db1155
  • db1136
  • db1127
  • db1124
  • db1116
  • db1101
  • db1098
  • db1079
  • clouddb1021
  • clouddb1018
  • clouddb1014
Marostegui updated the task description. (Show Details)Apr 27 2021, 5:40 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Apr 27 2021, 5:47 AM
Marostegui added a comment.EditedApr 27 2021, 5:52 AM

s3 eqiad

  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004
  • db1179
  • db1175
  • db1171
  • db1166
  • db1157
  • db1154
  • db1123
  • db1112
  • clouddb1021
  • clouddb1017
  • clouddb1013
Marostegui mentioned this in T278214: Switchover s1 from db1083 to db1163.Apr 27 2021, 6:03 AM
Marostegui added a comment.Apr 28 2021, 5:11 AM

Old s1 master (db1083) done

Marostegui updated the task description. (Show Details)Apr 28 2021, 5:11 AM
Marostegui updated the task description. (Show Details)Apr 28 2021, 5:32 AM
Marostegui added a comment.Apr 28 2021, 5:37 AM

All done

Marostegui closed this task as Resolved.Apr 28 2021, 5:37 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.May 5 2021, 9:41 PM

Change 685607 had a related patch set uploaded (by Ammarpad; author: Ammarpad):

[mediawiki/core@master] Use `rc_new_name_timestamp` index name unconditionally

https://gerrit.wikimedia.org/r/685607

gerritbot added a project: Patch-For-Review.May 5 2021, 9:41 PM
gerritbot added a comment.May 5 2021, 10:10 PM

Change 685607 merged by jenkins-bot:

[mediawiki/core@master] Use `rc_new_name_timestamp` index name unconditionally

https://gerrit.wikimedia.org/r/685607

Maintenance_bot removed a project: Patch-For-Review.May 5 2021, 10:11 PM
ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.5; 2021-05-11).May 5 2021, 11:00 PM
Umherirrender reopened this task as Open.Dec 22 2021, 6:34 PM
Umherirrender subscribed.

The "drift tracker" https://drift-tracker.toolforge.org/report/core/ reports one remaining database (server)

section	host	db
s2	db1156	bgwiki

It is list as done in T276292#7023496
Is that a problem of the drift tracker or something missed?

Marostegui added a comment.Dec 22 2021, 6:44 PM

Probably a race condition if the host was recloned in between

Umherirrender added a comment.Dec 22 2021, 6:47 PM
In T276292#7586215, @Marostegui wrote:

Probably a race condition if the host was recloned in between

Does that needs tracking? The index could be hinted by the exact name which can make queries failing.
Maybe there need to be done some extra backup/image copy after the alter to avoid restore/reclone from old schemas?

RhinosF1 subscribed.EditedDec 22 2021, 7:08 PM
In T276292#7586215, @Marostegui wrote:

Probably a race condition if the host was recloned in between

It was a brand new host only added to production that day.
(https://github.com/wikimedia/puppet/commit/ffab1c5e2ac0d637c610a51dda94fb506ceaa761#diff-ec14ede91bffe7d41df750cbfa5c8df0d34c8e6a811e7ddef3f731df82b599c4)

Umherirrender mentioned this in T298212: Research items from Drift tracker report (December 2021) and create specific tasks to fix the issues.Dec 22 2021, 7:52 PM
Marostegui added a comment.Jan 3 2022, 6:20 AM
In T276292#7586325, @RhinosF1 wrote:
In T276292#7586215, @Marostegui wrote:

Probably a race condition if the host was recloned in between

It was a brand new host only added to production that day.
(https://github.com/wikimedia/puppet/commit/ffab1c5e2ac0d637c610a51dda94fb506ceaa761#diff-ec14ede91bffe7d41df750cbfa5c8df0d34c8e6a811e7ddef3f731df82b599c4)

It was a brand new host, but the data is cloned from an existing one.

Marostegui closed this task as Resolved.Jan 3 2022, 6:23 AM
In T276292#7586195, @Umherirrender wrote:

The "drift tracker" https://drift-tracker.toolforge.org/report/core/ reports one remaining database (server)

section	host	db
s2	db1156	bgwiki

It is list as done in T276292#7023496
Is that a problem of the drift tracker or something missed?

Fixed.

Stashbot added a comment.Apr 29 2022, 4:53 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-29T16:53:42Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1156 (T276292)', diff saved to https://phabricator.wikimedia.org/P27081 and previous config saved to /var/cache/conftool/dbconfig/20220429-165333-ladsgroup.json

Ladsgroup added a comment.Apr 29 2022, 4:54 PM

Needed to be done on rest of db1156 (my bad). Fixing it now.

Stashbot added a comment.Apr 29 2022, 4:56 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-29T16:56:14Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T276292)', diff saved to https://phabricator.wikimedia.org/P27082 and previous config saved to /var/cache/conftool/dbconfig/20220429-165613-ladsgroup.json

Maintenance_bot moved this task from In progress to Done on the DBA board.Apr 29 2022, 5:29 PM
Stashbot added a comment.Apr 29 2022, 5:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-29T17:41:29Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T276292)', diff saved to https://phabricator.wikimedia.org/P27099 and previous config saved to /var/cache/conftool/dbconfig/20220429-174129-ladsgroup.json

Ladsgroup mentioned this in T307266: db1155 replication broken .Apr 30 2022, 7:46 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits