Page MenuHomePhabricator
Create Task
Maniphest T149079

codfw: Fix S4 commonswiki.templatelinks partitions
Closed, ResolvedPublic
Actions

Description

We have found that all the slaves have commonswiki. templatelinks table partitioned (including those servers which are not part of the rc service)

This is db2065

CREATE TABLE `templatelinks` (
  `tl_from` int(8) unsigned NOT NULL DEFAULT '0',
  `tl_namespace` int(11) NOT NULL DEFAULT '0',
  `tl_title` varbinary(255) NOT NULL DEFAULT '',
  `tl_from_namespace` int(11) NOT NULL DEFAULT '0',
  UNIQUE KEY `tl_from` (`tl_from`,`tl_namespace`,`tl_title`),
  KEY `tl_namespace` (`tl_namespace`,`tl_title`,`tl_from`),
  KEY `tl_backlinks_namespace` (`tl_from_namespace`,`tl_namespace`,`tl_title`,`tl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary
/*!50100 PARTITION BY RANGE (tl_namespace)
(PARTITION p_9 VALUES LESS THAN (10) ENGINE = InnoDB,
 PARTITION p_10 VALUES LESS THAN (11) ENGINE = InnoDB,
 PARTITION p_827 VALUES LESS THAN (828) ENGINE = InnoDB,
 PARTITION p_828 VALUES LESS THAN (829) ENGINE = InnoDB,
 PARTITION p_max VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */

After having a chat with @jcrespo looks like this is not intended and could be a mistake.

The slaves that need fixing are:

db2058
db2051
db2065
db2019 (master)

Details

SubjectRepoBranchLines +/-
db-eqiad.php: Depool db1064 for maintenanceoperations/mediawiki-configmaster+8 -6
db-eqiad.php: Depool db1068 for maintenanceoperations/mediawiki-configmaster+5 -4
db-eqiad.php: Repool db1059operations/mediawiki-configmaster+2 -2
db-eqiad.php: Depool db1059 for maintenanceoperations/mediawiki-configmaster+2 -2
Customize query in gerrit

Related Objects

Event Timeline

• Marostegui created this task.Oct 25 2016, 9:38 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2016, 9:38 AM
• Marostegui renamed this task from codfw: Fix S4 commonswiki.template links partitions to codfw: Fix S4 commonswiki.templatelinks partitions.Oct 25 2016, 9:38 AM
Stashbot subscribed.Oct 25 2016, 9:43 AM

Mentioned in SAL (#wikimedia-operations) [2016-10-25T09:43:07Z] <marostegui> Deploying ALTER table s4 commonswiki.templatelinks - T149079 (db2058 only)

• Marostegui added a comment.Oct 25 2016, 9:46 AM

This is now running - db2058:

./software/dbtools/osc_host.sh --host=db2058.codfw.wmnet --port=3306 --db=commonswiki --table=templatelinks --method=ddl --no-replicate "remove PARTITIONING"
• Marostegui moved this task from Triage to In progress on the DBA board.Oct 25 2016, 9:46 AM
• Marostegui mentioned this in T146261: Reimage dbstore2001 as jessie.Oct 25 2016, 11:01 AM
• Marostegui added a comment.Oct 26 2016, 5:42 AM

db2058 finished correctly

       Table: templatelinks
Create Table: CREATE TABLE `templatelinks` (
  `tl_from` int(8) unsigned NOT NULL DEFAULT '0',
  `tl_namespace` int(11) NOT NULL DEFAULT '0',
  `tl_title` varbinary(255) NOT NULL DEFAULT '',
  `tl_from_namespace` int(11) NOT NULL DEFAULT '0',
  UNIQUE KEY `tl_from` (`tl_from`,`tl_namespace`,`tl_title`),
  KEY `tl_namespace` (`tl_namespace`,`tl_title`,`tl_from`),
  KEY `tl_backlinks_namespace` (`tl_from_namespace`,`tl_namespace`,`tl_title`,`tl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary

root@db2058:/srv/sqldata/commonswiki# ls -lh templatelinks.*
-rw-rw---- 1 mysql mysql 2.3K Oct 25 10:52 templatelinks.frm
-rw-rw---- 1 mysql mysql 247G Oct 26 05:42 templatelinks.ibd
Stashbot added a comment.Oct 26 2016, 7:14 AM

Mentioned in SAL (#wikimedia-operations) [2016-10-26T07:14:33Z] <marostegui> Deploying ALTER table s4 commonswiki.templatelinks - db2051 - T149079

• Marostegui added a comment.Oct 26 2016, 7:15 AM

db2051 is now running the ALTER

• Marostegui added a comment.Oct 27 2016, 7:46 AM

db2051 finished:

root@neodymium:/home/marostegui/git# mysql -hdb2051.codfw.wmnet commonswiki -e "show create table templatelinks\G"
*************************** 1. row ***************************
       Table: templatelinks
Create Table: CREATE TABLE `templatelinks` (
  `tl_from` int(8) unsigned NOT NULL DEFAULT '0',
  `tl_namespace` int(11) NOT NULL DEFAULT '0',
  `tl_title` varbinary(255) NOT NULL DEFAULT '',
  `tl_from_namespace` int(11) NOT NULL DEFAULT '0',
  UNIQUE KEY `tl_from` (`tl_from`,`tl_namespace`,`tl_title`),
  KEY `tl_namespace` (`tl_namespace`,`tl_title`,`tl_from`),
  KEY `tl_backlinks_namespace` (`tl_from_namespace`,`tl_namespace`,`tl_title`,`tl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary
Stashbot added a comment.Oct 27 2016, 7:47 AM

Mentioned in SAL (#wikimedia-operations) [2016-10-27T07:47:12Z] <marostegui> Deploying ALTER table s4 commonswiki.templatelinks - https://phabricator.wikimedia.org/T149079 (db2065 only)

• Marostegui added a comment.Oct 27 2016, 7:52 AM

db2065 is now running.

I believe we'd also need to fix the following eqiad hosts:

db1059 (api)
db1064 (vslow, dump)
db1068 (api)
• Marostegui added a comment.Oct 27 2016, 5:18 PM

db2065 is now done

MariaDB MARIADB db2065.codfw.wmnet commonswiki > show create table templatelinks;
+---------------+----------------------------------------------------------------------------------------------------------------------
| Table         | Create Table
+---------------+----------------------------------------------------------------------------------------------------------------------
| templatelinks | CREATE TABLE `templatelinks` (
  `tl_from` int(8) unsigned NOT NULL DEFAULT '0',
  `tl_namespace` int(11) NOT NULL DEFAULT '0',
  `tl_title` varbinary(255) NOT NULL DEFAULT '',
  `tl_from_namespace` int(11) NOT NULL DEFAULT '0',
  UNIQUE KEY `tl_from` (`tl_from`,`tl_namespace`,`tl_title`),
  KEY `tl_namespace` (`tl_namespace`,`tl_title`,`tl_from`),
  KEY `tl_backlinks_namespace` (`tl_from_namespace`,`tl_namespace`,`tl_title`,`tl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary
• Marostegui added a comment.EditedOct 27 2016, 5:20 PM

The only pending one in codfw is the master (db2019) and then the other three servers in eqiad

• Marostegui added a comment.Nov 7 2016, 4:41 PM

Looks like removing partition isn't a online ddl operation.
Jaime and myself have agreed on not altering the master until we've done a failover to another host.

I will continue unifying the hosts in eqiad then

db1059 (api)
db1064 (vslow, dump)
db1068 (api)
gerritbot subscribed.Nov 8 2016, 7:28 AM

Change 320346 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Depool db1059 for maintenance

https://gerrit.wikimedia.org/r/320346

gerritbot added a project: Patch-For-Review.Nov 8 2016, 7:28 AM
gerritbot added a comment.Nov 8 2016, 10:06 AM

Change 320346 merged by jenkins-bot:
db-eqiad.php: Depool db1059 for maintenance

https://gerrit.wikimedia.org/r/320346

Stashbot added a comment.Nov 8 2016, 10:08 AM

Mentioned in SAL (#wikimedia-operations) [2016-11-08T10:08:04Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1059 - T149079 T147305 (duration: 00m 57s)

Stashbot mentioned this in T147305: Unify commonswiki.revision.Nov 8 2016, 10:08 AM
• Marostegui added a comment.Nov 8 2016, 4:41 PM

ALTER running on db1059.

• Marostegui added a comment.Nov 10 2016, 7:15 AM

db1059 is now finished

root@neodymium:~# mysql -hdb1059 -A commonswiki -e "nopager;show create table templatelinks\G"
PAGER set to stdout
*************************** 1. row ***************************
       Table: templatelinks
Create Table: CREATE TABLE `templatelinks` (
  `tl_from` int(8) unsigned NOT NULL DEFAULT '0',
  `tl_namespace` int(11) NOT NULL DEFAULT '0',
  `tl_title` varbinary(255) NOT NULL DEFAULT '',
  `tl_from_namespace` int(11) NOT NULL DEFAULT '0',
  UNIQUE KEY `tl_from` (`tl_from`,`tl_namespace`,`tl_title`),
  KEY `tl_namespace` (`tl_namespace`,`tl_title`,`tl_from`),
  KEY `tl_backlinks_namespace` (`tl_from_namespace`,`tl_namespace`,`tl_title`,`tl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary
gerritbot added a comment.Nov 10 2016, 7:19 AM

Change 320728 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Repool db1059

https://gerrit.wikimedia.org/r/320728

gerritbot added a comment.Nov 10 2016, 9:28 AM

Change 320728 merged by jenkins-bot:
db-eqiad.php: Repool db1059

https://gerrit.wikimedia.org/r/320728

Stashbot added a comment.Nov 10 2016, 9:38 AM

Mentioned in SAL (#wikimedia-operations) [2016-11-10T09:38:27Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: wmf-config/db-codfw.php Depool db1059 - T149079. Repool db2048 T150334 (duration: 00m 50s)

Stashbot mentioned this in T150334: db2042 reimage.Nov 10 2016, 9:38 AM
gerritbot added a comment.Nov 10 2016, 1:22 PM

Change 320773 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Depool db1068 for maintenance

https://gerrit.wikimedia.org/r/320773

gerritbot added a comment.Nov 10 2016, 2:02 PM

Change 320773 merged by jenkins-bot:
db-eqiad.php: Depool db1068 for maintenance

https://gerrit.wikimedia.org/r/320773

Stashbot added a comment.Nov 10 2016, 2:04 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-10T14:04:25Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1068 - T149079 (duration: 00m 48s)

Stashbot added a comment.Nov 10 2016, 3:09 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-10T15:09:28Z] <marostegui> Deploy schema change s4 commonswiki.template links (db1068) - https://phabricator.wikimedia.org/T149079

• Marostegui added a comment.Nov 10 2016, 3:17 PM

This is now running on db1068

• Marostegui added a comment.Nov 11 2016, 7:25 AM

db1068 is finished

MariaDB PRODUCTION s4 localhost commonswiki > show create table templatelinks\G
*************************** 1. row ***************************
       Table: templatelinks
Create Table: CREATE TABLE `templatelinks` (
  `tl_from` int(8) unsigned NOT NULL DEFAULT '0',
  `tl_namespace` int(11) NOT NULL DEFAULT '0',
  `tl_title` varbinary(255) NOT NULL DEFAULT '',
  `tl_from_namespace` int(11) NOT NULL DEFAULT '0',
  UNIQUE KEY `tl_from` (`tl_from`,`tl_namespace`,`tl_title`),
  KEY `tl_namespace` (`tl_namespace`,`tl_title`,`tl_from`),
  KEY `tl_backlinks_namespace` (`tl_from_namespace`,`tl_namespace`,`tl_title`,`tl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary
Stashbot added a comment.Nov 11 2016, 7:33 AM

Mentioned in SAL (#wikimedia-operations) [2016-11-11T07:33:08Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1068 - T149079 (duration: 00m 48s)

gerritbot added a comment.Nov 14 2016, 8:01 AM

Change 321354 had a related patch set uploaded (by Marostegui):
db-eqiad.php: Depool db1064 for maintenance

https://gerrit.wikimedia.org/r/321354

gerritbot added a comment.Nov 14 2016, 8:54 AM

Change 321354 merged by jenkins-bot:
db-eqiad.php: Depool db1064 for maintenance

https://gerrit.wikimedia.org/r/321354

Stashbot added a comment.Nov 14 2016, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2016-11-14T08:58:44Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1064 - T149079 (duration: 03m 07s)

Stashbot added a comment.Nov 14 2016, 9:11 AM

Mentioned in SAL (#wikimedia-operations) [2016-11-14T09:11:36Z] <marostegui> Deploy schema change s4 commonswiki templatelinks db1064 - T149079

• Marostegui added a comment.Nov 14 2016, 9:12 AM

This is now running in db1064, the last server to fix.

• Marostegui added a comment.Nov 15 2016, 7:24 AM

db1064 is finished:

root@neodymium:~# mysql -hdb1064 -A commonswiki -e "show create table templatelinks\G"
*************************** 1. row ***************************
       Table: templatelinks
Create Table: CREATE TABLE `templatelinks` (
  `tl_from` int(8) unsigned NOT NULL DEFAULT '0',
  `tl_namespace` int(11) NOT NULL DEFAULT '0',
  `tl_title` varbinary(255) NOT NULL DEFAULT '',
  `tl_from_namespace` int(11) NOT NULL DEFAULT '0',
  UNIQUE KEY `tl_from` (`tl_from`,`tl_namespace`,`tl_title`),
  KEY `tl_namespace` (`tl_namespace`,`tl_title`,`tl_from`),
  KEY `tl_backlinks_namespace` (`tl_from_namespace`,`tl_namespace`,`tl_title`,`tl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary
• Marostegui added a comment.Nov 15 2016, 9:35 AM

All the hosts are done, the only one pending is db2019 which is the master and will not be done as this operation isn't online as stated here: https://phabricator.wikimedia.org/T149079#2776840

Stashbot added a comment.Nov 15 2016, 9:43 AM

Mentioned in SAL (#wikimedia-operations) [2016-11-15T09:43:34Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1064 - T149079 (duration: 00m 51s)

• Marostegui added a parent task: T155099: Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover).Jan 11 2017, 5:42 PM
• Marostegui removed • Marostegui as the assignee of this task.Mar 3 2017, 3:58 PM
• Marostegui moved this task from In progress to Pending comment on the DBA board.
• Marostegui added a comment.Mar 31 2017, 7:41 AM

Will leave this alter table running on db2019 (codfw master) on Monday, it will generate lag on codfw.

• Marostegui removed a parent task: T155099: Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover).Mar 31 2017, 8:21 AM
• Marostegui added a comment.Apr 21 2017, 1:02 PM

This was done before the switchover

root@db2019.codfw.wmnet[commonswiki]> show create table templatelinks;
+---------------+----------------------------------------------------------------------------------------------------------------------
| Table         | Create Table
+---------------+----------------------------------------------------------------------------------------------------------------------
| templatelinks | CREATE TABLE `templatelinks` (
  `tl_from` int(8) unsigned NOT NULL DEFAULT '0',
  `tl_namespace` int(11) NOT NULL DEFAULT '0',
  `tl_title` varbinary(255) NOT NULL DEFAULT '',
  `tl_from_namespace` int(11) NOT NULL DEFAULT '0',
  UNIQUE KEY `tl_from` (`tl_from`,`tl_namespace`,`tl_title`),
  KEY `tl_namespace` (`tl_namespace`,`tl_title`,`tl_from`),
  KEY `tl_backlinks_namespace` (`tl_from_namespace`,`tl_namespace`,`tl_title`,`tl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary |
+---------------+----------------------------------------------------------------------------------------------------------------------
1 row in set (0.03 sec)
• Marostegui closed this task as Resolved.Apr 21 2017, 1:02 PM
• Marostegui claimed this task.
• Marostegui mentioned this in T258956: Remove commonswiki.templatelinks partitions.Jul 27 2020, 2:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL