Page MenuHomePhabricator
Create Task
Maniphest T237120

Schema change on production for increase the size of wbt_text_in_lang.wbxl_language
Closed, ResolvedPublic
Actions

Description

  1. ALTERs to run: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/547730/1/repo/sql/increase_wbxl_language.sql
  2. Where to run those changes: wikidatawiki (s8), testwikidatawiki (s3)
  3. When to run those changes: At any time, the sooner the better
  4. If the schema change is backwards compatible: Yes
  5. If the schema change has been tested already on some of the test/beta wikis: Tested in beta cluster
  6. if the data should be made available on the labs replicas and/or dumps: Yes, data in this table is public.

This blocks switching to read-new for the wb_terms migration.

Progress:

Related Objects
Search...

Event Timeline

Ladsgroup created this task.Nov 1 2019, 4:25 PM
jcrespo triaged this task as Medium priority.Nov 4 2019, 9:18 AM
jcrespo moved this task from Triage to Pending comment on the DBA board.

Thanks, acking this, but I hope this is not an emergency, as we may take a bit more than usual to apply it, as 50% of persistence team is on unavailable ATM. Would potentially starting next week be good enough (first on the queue)?

Ladsgroup added a comment.Nov 4 2019, 12:19 PM

Thanks. We got lucky and it's not a big blocker for us right now.

jcrespo assigned this task to • Marostegui.Nov 4 2019, 12:20 PM
• Addshore moved this task from incoming to in progress on the Wikidata board.Nov 5 2019, 2:10 PM
• Marostegui added a comment.Nov 11 2019, 11:01 AM

I can try to start with this next week to make sure the deletion of wb_terms doesn't get (more) blocked. I have a huge backlog to catch up with after my holidays so impossible to start with this during this week.
testwikidata should be easy to do and I will do that first (if that helps with anything)

Ladsgroup added a comment.Nov 11 2019, 11:37 AM
In T237120#5652812, @Marostegui wrote:

I can try to start with this next week to make sure the deletion of wb_terms doesn't get (more) blocked. I have a huge backlog to catch up with after my holidays so impossible to start with this during this week.
testwikidata should be easy to do and I will do that first (if that helps with anything)

I think it's fine. We have so many things to do, this is not a blocker right now.

• Marostegui moved this task from Backlog to In progress on the Schema-change-in-production board.Nov 18 2019, 9:13 AM
• Marostegui updated the task description. (Show Details)
• Marostegui moved this task from Pending comment to In progress on the DBA board.
Stashbot added a comment.Nov 18 2019, 10:56 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-18T10:56:28Z] <marostegui> Deploy schema change on db2078 (codfw master for wikidatawiki), this will create lag on s8 codfw - T237120

• Marostegui updated the task description. (Show Details)Nov 18 2019, 2:17 PM

This is how the table looks like after the ALTER:

       Table: wbt_text_in_lang
Create Table: CREATE TABLE `wbt_text_in_lang` (
  `wbxl_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `wbxl_language` varbinary(20) NOT NULL,
  `wbxl_text_id` int(10) unsigned NOT NULL,
  PRIMARY KEY (`wbxl_id`),
  UNIQUE KEY `wbt_text_in_lang_text_id_text_id` (`wbxl_text_id`,`wbxl_language`),
  KEY `wbt_text_in_lang_language` (`wbxl_language`)
) ENGINE=InnoDB AUTO_INCREMENT=328694252 DEFAULT CHARSET=binary
• Marostegui added a comment.EditedNov 18 2019, 2:23 PM

s8 eqiad progress

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1005
  • db1126
  • db1124
  • db1116
  • db1109 T237120#6432786
  • db1104
  • db1101
  • db1099
  • db1092
  • db1087
• Marostegui updated the task description. (Show Details)Nov 19 2019, 5:28 AM
Stashbot added a comment.Nov 20 2019, 7:14 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-20T07:14:00Z] <marostegui> Deploy schema change on s3 (testwikidatawiki) directly on s3 primary master T237120

• Marostegui updated the task description. (Show Details)Nov 20 2019, 7:14 AM
Stashbot mentioned this in T234066: Schema change to rename user_newtalk indexes.Nov 26 2019, 9:26 AM
Stashbot mentioned this in T233135: Schema change for refactored actor and comment storage.

Mentioned in SAL (#wikimedia-operations) [2019-11-26T09:26:50Z] <marostegui> Deploy schema change on s8 primary master (db1109) - T234066 T233135 T237120

• Marostegui added a comment.Nov 26 2019, 11:30 AM

Impossible to alter this table on the master while it is a master.
I just got a bunch of metadata locking for lots of connections and the cluster had a glitch. We'd need to schedule a master failover for s8 after Christmas in order to get the master done.

• Marostegui changed the task status from Open to Stalled.Nov 26 2019, 11:31 AM
• Marostegui updated the task description. (Show Details)
• Marostegui moved this task from In progress to Unclear/blocked to proceed on the Schema-change-in-production board.
• Marostegui moved this task from In progress to Blocked external/Not db team on the DBA board.
• Marostegui moved this task from in progress to blocked on others on the Wikidata board.
• Marostegui added a comment.Nov 26 2019, 11:38 AM

Probably a good idea to coordinate this required failover with: T238966 as that one will also probably require a failover as the revision table is quite big

• Marostegui mentioned this in T239238: Switchover s8 primary database master db1109 -> db1104 - 2020-09-29 08:00 UTC.Nov 26 2019, 2:18 PM
• Marostegui added a subtask: T239238: Switchover s8 primary database master db1109 -> db1104 - 2020-09-29 08:00 UTC.
• Marostegui added a comment.Jan 13 2020, 11:13 AM

We might be doing a DC switchover in Q3, so we can wait till eqiad is passive to address the final alter on the current primary master.

• Marostegui added a parent task: T243318: FY2020-2021 Q1 codfw -> eqiad switchback.Jan 22 2020, 6:12 AM
jcrespo changed the status of subtask T239238: Switchover s8 primary database master db1109 -> db1104 - 2020-09-29 08:00 UTC from Open to Stalled.Apr 29 2020, 7:30 AM
• Marostegui changed the status of subtask T239238: Switchover s8 primary database master db1109 -> db1104 - 2020-09-29 08:00 UTC from Stalled to Open.Aug 28 2020, 11:37 AM
• Marostegui changed the task status from Stalled to Open.Sep 3 2020, 5:41 AM
• Marostegui moved this task from Blocked external/Not db team to In progress on the DBA board.
• Marostegui moved this task from Unclear/blocked to proceed to In progress on the Schema-change-in-production board.
Stashbot added a comment.Sep 3 2020, 7:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T07:18:39Z] <marostegui> Stop slave on s8 eqiad master (lag will appear on s8 eqiad) - T237120

Stashbot added a comment.Sep 3 2020, 7:19 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-03T07:19:46Z] <marostegui> Deploy schema change on s8 eqiad master T237120

• Marostegui closed this task as Resolved.Sep 3 2020, 9:42 AM

The master is finally done

root@db1109.eqiad.wmnet[wikidatawiki]> ALTER TABLE /*_*/wbt_text_in_lang MODIFY wbxl_language VARBINARY(20) NOT NULL;
Query OK, 524052382 rows affected (1 hour 56 min 38.887 sec)
Records: 524052382  Duplicates: 0  Warnings: 0
• Marostegui updated the task description. (Show Details)Sep 3 2020, 9:42 AM
Kormat closed subtask T239238: Switchover s8 primary database master db1109 -> db1104 - 2020-09-29 08:00 UTC as Resolved.Sep 29 2020, 10:18 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL