Page MenuHomePhabricator

Change PK and remove partitions from the logging table
Closed, ResolvedPublic

Description

Special slaves are on the big wikis for the logging table an specific PK and partitions.
It has been tested that a normal slave (non partitioned) performs fine for the logpager and contributions service: T223151#5506269

There was just one query that was choosing the wrong index (most likely due to an optimizer misbehaviour) which required a workaround in code: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/538326/

The current table looks like:

CREATE TABLE `logging` (
  `log_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `log_type` varbinary(32) NOT NULL DEFAULT '',
  `log_action` varbinary(32) NOT NULL DEFAULT '',
  `log_timestamp` varbinary(14) NOT NULL DEFAULT '19700101000000',
  `log_user` int(10) unsigned NOT NULL DEFAULT '0',
  `log_namespace` int(11) NOT NULL DEFAULT '0',
  `log_title` varbinary(255) NOT NULL DEFAULT '',
  `log_comment` varbinary(255) NOT NULL DEFAULT '',
  `log_comment_id` bigint(20) unsigned NOT NULL DEFAULT '0',
  `log_params` blob NOT NULL,
  `log_deleted` tinyint(3) unsigned NOT NULL DEFAULT '0',
  `log_user_text` varbinary(255) NOT NULL DEFAULT '',
  `log_actor` bigint(20) unsigned NOT NULL DEFAULT '0',
  `log_page` int(10) unsigned DEFAULT NULL,
  PRIMARY KEY (`log_id`,`log_user`),
  KEY `type_time` (`log_type`,`log_timestamp`),
  KEY `user_time` (`log_user`,`log_timestamp`),
  KEY `page_time` (`log_namespace`,`log_title`,`log_timestamp`),
  KEY `times` (`log_timestamp`),
  KEY `log_page_id_time` (`log_page`,`log_timestamp`),
  KEY `log_user_type_time` (`log_user`,`log_type`,`log_timestamp`),
  KEY `actor_time` (`log_actor`,`log_timestamp`),
  KEY `log_actor_type_time` (`log_actor`,`log_type`,`log_timestamp`),
  KEY `log_type_action` (`log_type`,`log_action`,`log_timestamp`),
  KEY `log_user_text_type_time` (`log_user_text`,`log_type`,`log_timestamp`),
  KEY `log_user_text_time` (`log_user_text`,`log_timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=102192510 DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED
/*!50100 PARTITION BY RANGE (log_user)
(PARTITION p1 VALUES LESS THAN (1) ENGINE = InnoDB,
 PARTITION p50000 VALUES LESS THAN (50000) ENGINE = InnoDB,
 PARTITION p100000 VALUES LESS THAN (100000) ENGINE = InnoDB,
 PARTITION p200000 VALUES LESS THAN (200000) ENGINE = InnoDB,
 PARTITION p300000 VALUES LESS THAN (300000) ENGINE = InnoDB,
 PARTITION p400000 VALUES LESS THAN (400000) ENGINE = InnoDB,
 PARTITION p500000 VALUES LESS THAN (500000) ENGINE = InnoDB,
 PARTITION p750000 VALUES LESS THAN (750000) ENGINE = InnoDB,
 PARTITION p1000000 VALUES LESS THAN (1000000) ENGINE = InnoDB,
 PARTITION p1250000 VALUES LESS THAN (1250000) ENGINE = InnoDB,
 PARTITION p1500000 VALUES LESS THAN (1500000) ENGINE = InnoDB,
 PARTITION p1750000 VALUES LESS THAN (1750000) ENGINE = InnoDB,
 PARTITION p2000000 VALUES LESS THAN (2000000) ENGINE = InnoDB,
 PARTITION p2500000 VALUES LESS THAN (2500000) ENGINE = InnoDB,
 PARTITION p3000000 VALUES LESS THAN (3000000) ENGINE = InnoDB,
 PARTITION p3500000 VALUES LESS THAN (3500000) ENGINE = InnoDB,
 PARTITION p4000000 VALUES LESS THAN (4000000) ENGINE = InnoDB,
 PARTITION p4500000 VALUES LESS THAN (4500000) ENGINE = InnoDB,
 PARTITION p5000000 VALUES LESS THAN (5000000) ENGINE = InnoDB,
 PARTITION p6000000 VALUES LESS THAN (6000000) ENGINE = InnoDB,
 PARTITION p7000000 VALUES LESS THAN (7000000) ENGINE = InnoDB,
 PARTITION p7500000 VALUES LESS THAN (7500000) ENGINE = InnoDB,
 PARTITION p8000000 VALUES LESS THAN (8000000) ENGINE = InnoDB,
 PARTITION p9000000 VALUES LESS THAN (9000000) ENGINE = InnoDB,
 PARTITION p10000000 VALUES LESS THAN (10000000) ENGINE = InnoDB,
 PARTITION p11000000 VALUES LESS THAN (11000000) ENGINE = InnoDB,
 PARTITION p12000000 VALUES LESS THAN (12000000) ENGINE = InnoDB,
 PARTITION p13000000 VALUES LESS THAN (13000000) ENGINE = InnoDB,
 PARTITION p14000000 VALUES LESS THAN (14000000) ENGINE = InnoDB,
 PARTITION p16000000 VALUES LESS THAN (16000000) ENGINE = InnoDB,
 PARTITION p18000000 VALUES LESS THAN (18000000) ENGINE = InnoDB,
 PARTITION p22000000 VALUES LESS THAN (22000000) ENGINE = InnoDB,
 PARTITION p24000000 VALUES LESS THAN (24000000) ENGINE = InnoDB,
 PARTITION p28000000 VALUES LESS THAN (28000000) ENGINE = InnoDB,
 PARTITION pMAXVALUE VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */

And it should look like:

CREATE TABLE `logging` (
  `log_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `log_type` varbinary(32) NOT NULL DEFAULT '',
  `log_action` varbinary(32) NOT NULL DEFAULT '',
  `log_timestamp` varbinary(14) NOT NULL DEFAULT '19700101000000',
  `log_user` int(10) unsigned NOT NULL DEFAULT '0',
  `log_namespace` int(11) NOT NULL DEFAULT '0',
  `log_title` varbinary(255) NOT NULL DEFAULT '',
  `log_comment` varbinary(255) NOT NULL DEFAULT '',
  `log_comment_id` bigint(20) unsigned NOT NULL DEFAULT '0',
  `log_params` blob NOT NULL,
  `log_deleted` tinyint(3) unsigned NOT NULL DEFAULT '0',
  `log_user_text` varbinary(255) NOT NULL DEFAULT '',
  `log_actor` bigint(20) unsigned NOT NULL DEFAULT '0',
  `log_page` int(10) unsigned DEFAULT NULL,
  PRIMARY KEY (`log_id`),
  KEY `type_time` (`log_type`,`log_timestamp`),
  KEY `user_time` (`log_user`,`log_timestamp`),
  KEY `page_time` (`log_namespace`,`log_title`,`log_timestamp`),
  KEY `times` (`log_timestamp`),
  KEY `log_page_id_time` (`log_page`,`log_timestamp`),
  KEY `log_user_text_time` (`log_user_text`(16),`log_timestamp`),
  KEY `log_user_type_time` (`log_user`,`log_type`,`log_timestamp`),
  KEY `actor_time` (`log_actor`,`log_timestamp`),
  KEY `log_actor_type_time` (`log_actor`,`log_type`,`log_timestamp`),
  KEY `log_type_action` (`log_type`,`log_action`,`log_timestamp`),
  KEY `log_user_text_type_time` (`log_user_text`,`log_type`,`log_timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=102192535 DEFAULT CHARSET=binary

Once that code fix is deployed, the special slaves on big wikis on codfw should get the following alters:

ALTER TABLE logging REMOVE PARTITIONING;
ALTER TABLE logging DROP PRIMARY KEY, ADD PRIMARY KEY (log_id);
ALTER TABLE logging ROW_FORMAT=COMPRESSED, KEY_BLOCK_SIZE=8, ALGORITHM=COPY;
  • s1
    • db1099 - table compressed
    • db1105 - table compressed
    • db2085 - table compressed
    • db2088 - table compressed
  • s2
    • db1103 - table compressed
    • db1105 - table compressed
    • db2088 - table compressed
    • db2091 - table compressed
  • s3 - no partitions
  • s4
    • db1097 - table compressed
    • db1103 - table compressed
    • db2084 - table compressed
    • db2091 - table compressed
  • s5
    • db1096 - table compressed
    • db1097 - table compressed
    • db2084 - table compressed
    • db2089 - table compressed
  • s6
    • db1096 - table compressed
    • db1098 - table compressed
    • db2087 - table compressed
    • db2089 - table compressed
  • s7
    • db1098 - table compressed
    • db1101 - table compressed
    • db2086 - table compressed
    • db2087 - table compressed
  • s8
    • db1099 - table compressed
    • db1101 - table compressed
    • db2085 - table compressed
    • db2086 - table compressed

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2019-09-25T06:20:37Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2085:3311 T233625', diff saved to https://phabricator.wikimedia.org/P9171 and previous config saved to /var/cache/conftool/dbconfig/20190925-062036-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-25T06:21:09Z] <marostegui> Deploy schema change on db2085:3311 T233625

db2085 enwiki is done:

root@cumin1001:/home/marostegui/T233625# mysql.py -hdb2085:3311 enwiki -e "show create table logging\G"
*************************** 1. row ***************************
       Table: logging
Create Table: CREATE TABLE `logging` (
  `log_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `log_type` varbinary(32) NOT NULL DEFAULT '',
  `log_action` varbinary(32) NOT NULL DEFAULT '',
  `log_timestamp` varbinary(14) NOT NULL DEFAULT '19700101000000',
  `log_user` int(10) unsigned NOT NULL DEFAULT '0',
  `log_namespace` int(11) NOT NULL DEFAULT '0',
  `log_title` varbinary(255) NOT NULL DEFAULT '',
  `log_comment` varbinary(255) NOT NULL DEFAULT '',
  `log_comment_id` bigint(20) unsigned NOT NULL DEFAULT '0',
  `log_params` blob NOT NULL,
  `log_deleted` tinyint(3) unsigned NOT NULL DEFAULT '0',
  `log_user_text` varbinary(255) NOT NULL DEFAULT '',
  `log_actor` bigint(20) unsigned NOT NULL DEFAULT '0',
  `log_page` int(10) unsigned DEFAULT NULL,
  PRIMARY KEY (`log_id`),
  KEY `type_time` (`log_type`,`log_timestamp`),
  KEY `user_time` (`log_user`,`log_timestamp`),
  KEY `page_time` (`log_namespace`,`log_title`,`log_timestamp`),
  KEY `times` (`log_timestamp`),
  KEY `log_page_id_time` (`log_page`,`log_timestamp`),
  KEY `log_user_type_time` (`log_user`,`log_type`,`log_timestamp`),
  KEY `actor_time` (`log_actor`,`log_timestamp`),
  KEY `log_actor_type_time` (`log_actor`,`log_type`,`log_timestamp`),
  KEY `log_type_action` (`log_type`,`log_action`,`log_timestamp`),
  KEY `log_user_text_type_time` (`log_user_text`,`log_type`,`log_timestamp`),
  KEY `log_user_text_time` (`log_user_text`,`log_timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=102239931 DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8

And it now looks like a non special host in enwiki:

mysql.py -h db1089 enwiki -e "show create table logging\G"
*************************** 1. row ***************************
       Table: logging
Create Table: CREATE TABLE `logging` (
  `log_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `log_type` varbinary(32) NOT NULL DEFAULT '',
  `log_action` varbinary(32) NOT NULL DEFAULT '',
  `log_timestamp` varbinary(14) NOT NULL DEFAULT '19700101000000',
  `log_user` int(10) unsigned NOT NULL DEFAULT '0',
  `log_namespace` int(11) NOT NULL DEFAULT '0',
  `log_title` varbinary(255) NOT NULL DEFAULT '',
  `log_comment` varbinary(255) NOT NULL DEFAULT '',
  `log_comment_id` bigint(20) unsigned NOT NULL DEFAULT '0',
  `log_params` blob NOT NULL,
  `log_deleted` tinyint(3) unsigned NOT NULL DEFAULT '0',
  `log_user_text` varbinary(255) NOT NULL DEFAULT '',
  `log_actor` bigint(20) unsigned NOT NULL DEFAULT '0',
  `log_page` int(10) unsigned DEFAULT NULL,
  PRIMARY KEY (`log_id`),
  KEY `type_time` (`log_type`,`log_timestamp`),
  KEY `user_time` (`log_user`,`log_timestamp`),
  KEY `page_time` (`log_namespace`,`log_title`,`log_timestamp`),
  KEY `times` (`log_timestamp`),
  KEY `log_page_id_time` (`log_page`,`log_timestamp`),
  KEY `log_user_text_time` (`log_user_text`(16),`log_timestamp`),
  KEY `log_user_type_time` (`log_user`,`log_type`,`log_timestamp`),
  KEY `actor_time` (`log_actor`,`log_timestamp`),
  KEY `log_actor_type_time` (`log_actor`,`log_type`,`log_timestamp`),
  KEY `log_type_action` (`log_type`,`log_action`,`log_timestamp`),
  KEY `log_user_text_type_time` (`log_user_text`,`log_type`,`log_timestamp`)
) ENGINE=InnoDB AUTO_INCREMENT=102241030 DEFAULT CHARSET=binary

Mentioned in SAL (#wikimedia-operations) [2019-09-25T12:51:42Z] <marostegui@cumin1001> dbctl commit (dc=all): ' Depool for schema change on the logging table: db2088:3312 db2084:3315 db2087:3316 db2086:3317 T233625', diff saved to https://phabricator.wikimedia.org/P9177 and previous config saved to /var/cache/conftool/dbconfig/20190925-125140-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-25T12:56:02Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2085:3311 T233625', diff saved to https://phabricator.wikimedia.org/P9178 and previous config saved to /var/cache/conftool/dbconfig/20190925-125601-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T06:29:23Z] <marostegui@cumin1001> dbctl commit (dc=all): ' Repool db2088:3312 db2084:3315 db2087:3316 db2086:3317 T233625', diff saved to https://phabricator.wikimedia.org/P9195 and previous config saved to /var/cache/conftool/dbconfig/20190926-062922-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T06:35:56Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2088:3311 db2091:3312 db2084:3314 db2089:3315 db2089:3316 db2087:3317 T233625', diff saved to https://phabricator.wikimedia.org/P9196 and previous config saved to /var/cache/conftool/dbconfig/20190926-063555-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-09-26T06:39:52Z] <marostegui> Deploy schema change on db2088:3311 db2091:3312 db2084:3314 db2089:3315 db2089:3316 db2087:3317 T233625

Change 539319 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] sX-pager.sql: Remove partitioning from logging table

https://gerrit.wikimedia.org/r/539319

Change 539319 merged by jenkins-bot:
[operations/software@master] sX-pager.sql: Remove partitioning from logging table

https://gerrit.wikimedia.org/r/539319

Mentioned in SAL (#wikimedia-operations) [2019-09-27T10:49:15Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2088:3311 db2091:3312 db2084:3314 db2089:3315 db2089:3316 db2087:3317 T233625', diff saved to https://phabricator.wikimedia.org/P9213 and previous config saved to /var/cache/conftool/dbconfig/20190927-104914-marostegui.json

I think the reason that it doesn't need partitioning anymore is T184485: Stop logging autopatrol actions (and things like T189596: Run deleteAutopatrolLogs script for Wikidata (WMF)) which dropped 1.4B rows from logging tables. (Exact numbers). What I want to say is that I think it's fine to de-partition it.

I think the reason that it doesn't need partitioning anymore is T184485: Stop logging autopatrol actions (and things like T189596: Run deleteAutopatrolLogs script for Wikidata (WMF)) which dropped 1.4B rows from logging tables. (Exact numbers). What I want to say is that I think it's fine to de-partition it.

<3 <3 <3

Mentioned in SAL (#wikimedia-operations) [2019-09-30T09:10:45Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2091:3314 for a schema change - T233625', diff saved to https://phabricator.wikimedia.org/P9217 and previous config saved to /var/cache/conftool/dbconfig/20190930-091043-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-01T06:19:57Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2091:3314 schema change - T233625', diff saved to https://phabricator.wikimedia.org/P9223 and previous config saved to /var/cache/conftool/dbconfig/20191001-061956-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-03T08:28:58Z] <marostegui> Deploy schema change on db1096:3316 - T233625

Mentioned in SAL (#wikimedia-operations) [2019-10-07T06:56:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096:3315 for schema change T233625', diff saved to https://phabricator.wikimedia.org/P9246 and previous config saved to /var/cache/conftool/dbconfig/20191007-065645-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-07T13:03:19Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1096:3315 after schema change T233625', diff saved to https://phabricator.wikimedia.org/P9247 and previous config saved to /var/cache/conftool/dbconfig/20191007-130317-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-08T05:07:44Z] <marostegui> Deploy schema change on db1097:3315 - T233625

Mentioned in SAL (#wikimedia-operations) [2019-10-08T05:08:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3315 T233625', diff saved to https://phabricator.wikimedia.org/P9252 and previous config saved to /var/cache/conftool/dbconfig/20191008-050833-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-08T09:06:17Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1097:3315 T233625', diff saved to https://phabricator.wikimedia.org/P9257 and previous config saved to /var/cache/conftool/dbconfig/20191008-090616-marostegui.json

We have to compress also the logging table once it gets its partitioning removed.

Mentioned in SAL (#wikimedia-operations) [2019-10-08T13:50:59Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Depool db1103:3312 for schema change T233625', diff saved to https://phabricator.wikimedia.org/P9266 and previous config saved to /var/cache/conftool/dbconfig/20191008-135058-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-09T12:56:43Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1099:3318 for schema change T233625', diff saved to https://phabricator.wikimedia.org/P9279 and previous config saved to /var/cache/conftool/dbconfig/20191009-125641-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-09T14:11:38Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1099:3318 after schema change T233625', diff saved to https://phabricator.wikimedia.org/P9283 and previous config saved to /var/cache/conftool/dbconfig/20191009-141137-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-09T14:46:08Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3318 for schema change T233625', diff saved to https://phabricator.wikimedia.org/P9285 and previous config saved to /var/cache/conftool/dbconfig/20191009-144607-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-11T04:54:11Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098:3317 for schema change T233625', diff saved to https://phabricator.wikimedia.org/P9310 and previous config saved to /var/cache/conftool/dbconfig/20191011-045409-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-11T12:32:00Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1098:3317 after schema change T233625', diff saved to https://phabricator.wikimedia.org/P9314 and previous config saved to /var/cache/conftool/dbconfig/20191011-123159-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-14T04:56:31Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3317 for schema change T233625', diff saved to https://phabricator.wikimedia.org/P9318 and previous config saved to /var/cache/conftool/dbconfig/20191014-045629-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-14T14:28:44Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1103:3314 for schema change T233625', diff saved to https://phabricator.wikimedia.org/P9329 and previous config saved to /var/cache/conftool/dbconfig/20191014-142843-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2019-10-15T05:28:29Z] <marostegui> Deploy schema change on db1097:3314 T233625

Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)

All partitions removed from the logging table of all the special slaves.