db1213 replication broken (Index for table dewiki.page_props is corrupt)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	Jun 2 2024, 8:50 PM

Description

<+icinga-wm_> PROBLEM - MariaDB Replica SQL: s5 #page on db1213 is CRITICAL: CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1034, Errmsg: Error Index for table page_props is corrupt: try to repair it on query. Default database: dewiki. [Query snipped] https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica

<+logmsgbot> !log taavi@cumin1002 dbctl commit (dc=all): 'depool db1213', diff saved to https://phabricator.wikimedia.org/P63893 and previous config saved to /var/cache/conftool/dbconfig/20240602-204719-taavi.json

Related Objects

Mentioned Here: P63912 dbctl commit (dc=all): 'db1213 (re)pooling @ 100%: Repooling T366429'
P63910 dbctl commit (dc=all): 'db1213 (re)pooling @ 75%: Repooling T366429'
P63908 dbctl commit (dc=all): 'db1213 (re)pooling @ 50%: Repooling T366429'
P63907 dbctl commit (dc=all): 'db1213 (re)pooling @ 25%: Repooling T366429'
P63906 dbctl commit (dc=all): 'db1213 (re)pooling @ 10%: Repooling T366429'
P63905 dbctl commit (dc=all): 'db1213 (re)pooling @ 5%: Repooling T366429'
P63904 dbctl commit (dc=all): 'db1213 (re)pooling @ 1%: Repooling T366429'

Event Timeline

taavi created this task.Jun 2 2024, 8:50 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 2 2024, 8:50 PM

taavi added a project: DBA.Jun 2 2024, 8:51 PM

• OKJ04 added a subtask: T366433: CentralAuth tests broken unless you run them inside Quibble.Jun 3 2024, 1:39 AM

JJMC89 removed a subtask: T366433: CentralAuth tests broken unless you run them inside Quibble.Jun 3 2024, 1:44 AM

Is there any schema change going on on this table?
I have rebuilt the table and replication is back up.

Host back in sync with the master, I am going remove downtime and start repooling.

Mentioned in SAL (#wikimedia-operations) [2024-06-03T04:49:18Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 1%: Repooling T366429', diff saved to https://phabricator.wikimedia.org/P63904 and previous config saved to /var/cache/conftool/dbconfig/20240603-044918-root.json

Marostegui closed this task as Resolved.Jun 3 2024, 4:59 AM

Mentioned in SAL (#wikimedia-operations) [2024-06-03T05:04:26Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 5%: Repooling T366429', diff saved to https://phabricator.wikimedia.org/P63905 and previous config saved to /var/cache/conftool/dbconfig/20240603-050424-root.json

Mentioned in SAL (#wikimedia-operations) [2024-06-03T05:19:33Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 10%: Repooling T366429', diff saved to https://phabricator.wikimedia.org/P63906 and previous config saved to /var/cache/conftool/dbconfig/20240603-051932-root.json

Maintenance_bot moved this task from In progress to Done on the DBA board.Jun 3 2024, 5:29 AM

Mentioned in SAL (#wikimedia-operations) [2024-06-03T05:34:39Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 25%: Repooling T366429', diff saved to https://phabricator.wikimedia.org/P63907 and previous config saved to /var/cache/conftool/dbconfig/20240603-053438-root.json

Mentioned in SAL (#wikimedia-operations) [2024-06-03T05:49:45Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 50%: Repooling T366429', diff saved to https://phabricator.wikimedia.org/P63908 and previous config saved to /var/cache/conftool/dbconfig/20240603-054944-root.json

Mentioned in SAL (#wikimedia-operations) [2024-06-03T06:04:51Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 75%: Repooling T366429', diff saved to https://phabricator.wikimedia.org/P63910 and previous config saved to /var/cache/conftool/dbconfig/20240603-060450-root.json

Mentioned in SAL (#wikimedia-operations) [2024-06-03T06:19:57Z] <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 100%: Repooling T366429', diff saved to https://phabricator.wikimedia.org/P63912 and previous config saved to /var/cache/conftool/dbconfig/20240603-061956-root.json

db1213 replication broken (Index for table dewiki.page_props is corrupt)Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

db1213 replication broken (Index for table dewiki.page_props is corrupt)
Closed, ResolvedPublic
Actions