Page MenuHomePhabricator

db2182 depooled (Errmsg: Error Index for table recentchanges is corrupt)
Closed, ResolvedPublic

Description

2025-01-26 21:33:17 7 [ERROR] Slave SQL: Error 'Index for table 'recentchanges' is corrupt; try to repair it' on query. Default database: 'rowiki'. Query: 'INSERT /* RecentChange::save  */ INTO `recentchanges`

Details

Related Changes in Gerrit:

Event Timeline

Icinga downtime and Alertmanager silence (ID=54251e02-6931-4d41-b42e-c1e9b933f135) set by mvernon@cumin2002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: needs index repair

db2182.codfw.wmnet

Change #1114258 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2182: Disable notifications

https://gerrit.wikimedia.org/r/1114258

Change #1114258 merged by Marostegui:

[operations/puppet@production] db2182: Disable notifications

https://gerrit.wikimedia.org/r/1114258

Marostegui triaged this task as Medium priority.Jan 27 2025, 7:42 AM
Marostegui moved this task from Triage to In progress on the DBA board.

The host was upgraded and the tables are now being rebuilt.

Tables rebuilt, host catching up.

Mentioned in SAL (#wikimedia-operations) [2025-01-27T13:15:54Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 10%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72479 and previous config saved to /var/cache/conftool/dbconfig/20250127-131554-root.json

Host being repooled automatically.

Mentioned in SAL (#wikimedia-operations) [2025-01-27T13:31:01Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 25%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72482 and previous config saved to /var/cache/conftool/dbconfig/20250127-133059-root.json

Mentioned in SAL (#wikimedia-operations) [2025-01-27T13:46:06Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 50%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72483 and previous config saved to /var/cache/conftool/dbconfig/20250127-134605-root.json

Mentioned in SAL (#wikimedia-operations) [2025-01-27T14:01:11Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 75%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72485 and previous config saved to /var/cache/conftool/dbconfig/20250127-140111-root.json

Mentioned in SAL (#wikimedia-operations) [2025-01-27T14:16:17Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db2182 (re)pooling @ 100%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72487 and previous config saved to /var/cache/conftool/dbconfig/20250127-141616-root.json