Was depooled and downtimed until Monday.
CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1034, Errmsg: Error 'Index for table 'recentchanges' is corrupt: try to repair it' on query. Default database: 'nlwiki'. [Query snipped]
Description
Description
Related Objects
Related Objects
- Mentioned Here
- P69695 dbctl commit (dc=all): 'db2175 (re)pooling @ 100%: T376988'
P69694 dbctl commit (dc=all): 'db2175 (re)pooling @ 75%: T376988'
P69693 dbctl commit (dc=all): 'db2175 (re)pooling @ 50%: T376988'
P69692 dbctl commit (dc=all): 'db2175 (re)pooling @ 25%: T376988'
P69691 dbctl commit (dc=all): 'db2175 (re)pooling @ 10%: T376988'
P69690 dbctl commit (dc=all): 'db2175 (re)pooling @ 5%: T376988'
P69688 dbctl commit (dc=all): 'db2175 (re)pooling @ 2%: T376988'
P69687 dbctl commit (dc=all): 'db2175 (re)pooling @ 1%: T376988'
Event Timeline
Comment Actions
Started on a screen on db2175:
mysql:root@localhost [nlwiki]> ALTER TABLE recentchanges ENGINE=InnoDB, FORCE;
Comment Actions
The repair finished in 10 seconds, faster than I expected without issues (table & index rebuild succesfully, and I restarted replication).
@ABran-WMF When you are back, please check everything is fine (the server, the table, replication, no other strange logs), the right procedure is followed, and specially the host is removed the long downtime and repooled, in that order. If happy, then please resolve the ticket.
Comment Actions
Not needed to close the ticket- but I found something really weird- no megacli cmd. Do you know what's the binary for manual checking on newer hosts? Also, learning cycles are disabled on those hosts, too, right?
Comment Actions
Please add it there, I am happy to help with the mysql side, but I am not familiar with the DBA's day to day procedures.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-10-11T12:13:26Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2175 (re)pooling @ 1%: T376988', diff saved to https://phabricator.wikimedia.org/P69687 and previous config saved to /var/cache/conftool/dbconfig/20241011-121325-arnaudb.json
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-10-11T12:28:31Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2175 (re)pooling @ 2%: T376988', diff saved to https://phabricator.wikimedia.org/P69688 and previous config saved to /var/cache/conftool/dbconfig/20241011-122830-arnaudb.json
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-10-11T12:43:36Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2175 (re)pooling @ 5%: T376988', diff saved to https://phabricator.wikimedia.org/P69690 and previous config saved to /var/cache/conftool/dbconfig/20241011-124336-arnaudb.json
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-10-11T12:58:42Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2175 (re)pooling @ 10%: T376988', diff saved to https://phabricator.wikimedia.org/P69691 and previous config saved to /var/cache/conftool/dbconfig/20241011-125841-arnaudb.json
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-10-11T13:13:47Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2175 (re)pooling @ 25%: T376988', diff saved to https://phabricator.wikimedia.org/P69692 and previous config saved to /var/cache/conftool/dbconfig/20241011-131347-arnaudb.json
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-10-11T13:28:53Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2175 (re)pooling @ 50%: T376988', diff saved to https://phabricator.wikimedia.org/P69693 and previous config saved to /var/cache/conftool/dbconfig/20241011-132852-arnaudb.json
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-10-11T13:43:59Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2175 (re)pooling @ 75%: T376988', diff saved to https://phabricator.wikimedia.org/P69694 and previous config saved to /var/cache/conftool/dbconfig/20241011-134357-arnaudb.json
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-10-11T13:59:04Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db2175 (re)pooling @ 100%: T376988', diff saved to https://phabricator.wikimedia.org/P69695 and previous config saved to /var/cache/conftool/dbconfig/20241011-135903-arnaudb.json