Page MenuHomePhabricator

db1233 ptwiki.page_props index corrupted
Closed, ResolvedPublic

Description

Last_SQL_Error: Error 'Index for table 'page_props' is corrupt; try to repair it' on query. Default database: 'ptwiki'>

Event Timeline

This table is in a very bad state looks like:

cumin2024@db1233.eqiad.wmnet[ptwiki]> alter table page_props engine=Innodb,force;
ERROR 1062 (23000): Duplicate entry 'defaultsort-7487099' for key 'pp_propname_page'

However:

cumin2024@db1233.eqiad.wmnet[ptwiki]> SELECT * FROM page_props WHERE pp_propname = 'defaultsort' AND pp_page = 7487099;
+---------+-------------+---------------+------------+
| pp_page | pp_propname | pp_value      | pp_sortkey |
+---------+-------------+---------------+------------+
| 7487099 | defaultsort | Soares, Juary |       NULL |
+---------+-------------+---------------+------------+
1 row in set (0.001 sec)

Yeah, this table is basically broken. It is also showing duplicate key for the other unique, and there are not really any duplicate records there. @Ladsgroup I am not sure if MW has something to do with this. There is definitely no duplicate entries on this table for that value.
I've removed both unique keys in case they were both corrupt and showing bad data when queried, but without indexes on the table, the query is the same:

| page_props | CREATE TABLE `page_props` (
  `pp_page` int(10) unsigned NOT NULL,
  `pp_propname` varbinary(60) NOT NULL DEFAULT '',
  `pp_value` blob NOT NULL,
  `pp_sortkey` float DEFAULT NULL,
  PRIMARY KEY (`pp_page`,`pp_propname`)
) ENGINE=InnoDB DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8 |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.001 sec)

cumin2024@db1233.eqiad.wmnet[ptwiki]> SELECT * FROM page_props WHERE pp_propname = 'defaultsort' AND pp_page = 7487099;
+---------+-------------+---------------+------------+
| pp_page | pp_propname | pp_value      | pp_sortkey |
+---------+-------------+---------------+------------+
| 7487099 | defaultsort | Soares, Juary |       NULL |
+---------+-------------+---------------+------------+
1 row in set (0.001 sec)

cumin2024@db1233.eqiad.wmnet[ptwiki]> alter table page_props add unique index pp_propname_page (pp_propname, pp_page);
ERROR 1062 (23000): Duplicate entry 'defaultsort-7487099' for key 'pp_propname_page'

I've tried to dump the table and placed it on the test cluster and see if it would go thru clean, but it doesn't - so for now dumping and importing the table wouldn't work.

ERROR 1062 (23000) at line 210: Duplicate entry '7486806-wikibase_item' for key 'PRIMARY'

All this time isn't worth it, let's just reclone this host.

Mentioned in SAL (#wikimedia-operations) [2024-12-01T10:44:41Z] <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depool to reclone (T381213)', diff saved to https://phabricator.wikimedia.org/P71451 and previous config saved to /var/cache/conftool/dbconfig/20241201-104441-ladsgroup.json

I picked the sanitarium master because it had one of the lowest weights (but I didn't know it's santarium master). oh well.

Mentioned in SAL (#wikimedia-operations) [2024-12-01T12:16:40Z] <ladsgroup@cumin1002> START - Cookbook sre.mysql.pool db1233 gradually with 4 steps - Maint over (T381213)

Mentioned in SAL (#wikimedia-operations) [2024-12-01T12:31:48Z] <ladsgroup@cumin1002> START - Cookbook sre.mysql.pool db1156 gradually with 4 steps - Maint over (T381213)

Ladsgroup moved this task from In progress to Done on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2024-12-01T13:02:03Z] <ladsgroup@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1233 gradually with 4 steps - Maint over (T381213)

Mentioned in SAL (#wikimedia-operations) [2024-12-01T13:17:43Z] <ladsgroup@cumin1002> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1156 gradually with 4 steps - Maint over (T381213)