db1169 replication broken - enwiki.pagelinks corruption
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	Oct 28 2024, 6:04 AM

Description

20:56 <+icinga-wm> PROBLEM - MariaDB Replica SQL: s1 on db1169 is CRITICAL: CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1034, Errmsg: Error Index for table pagelinks is corrupt: try to repair it on query. Default database: enwiki. [Query snipped] https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
21:04 PROBLEM - MariaDB Replica Lag: s1 on db1169 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 656.04 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica

I depooled it since it was causing issues with bots due to maxlag going up: P70590

Related Objects

Mentioned In: T378311: Refill tool stuck "waiting for an available worker"
Mentioned Here: P70604 dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: T378320'
P70603 dbctl commit (dc=all): 'db1169 (re)pooling @ 75%: T378320'
P70600 dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: T378320'
P70599 dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: T378320'
P70590 dbctl commit (dc=all): 'depool db1169'

Event Timeline

taavi created this task.Oct 28 2024, 6:04 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2024, 6:04 AM

Mentioned in SAL (#wikimedia-operations) [2024-10-28T06:06:55Z] <taavi@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: replication broken T378320

Mentioned in SAL (#wikimedia-operations) [2024-10-28T06:07:09Z] <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: replication broken T378320

Mdann52 mentioned this in T378311: Refill tool stuck "waiting for an available worker".Oct 28 2024, 6:07 AM

ABran-WMF claimed this task.Oct 28 2024, 9:35 AM

ABran-WMF triaged this task as High priority.

ABran-WMF moved this task from Triage to In progress on the DBA board.

index rebuilt: Query OK, 0 rows affected (2 hours 44 min 43.595 sec)
replication catching back up

Start pool of db1169 gradually with 4 steps - index rebuilt - arnaudb@cumin1002

Start pool of db1169 quickly with 2 steps - index rebuilt - arnaudb@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-10-29T08:09:52Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: T378320', diff saved to https://phabricator.wikimedia.org/P70599 and previous config saved to /var/cache/conftool/dbconfig/20241029-080951-arnaudb.json

ABran-WMF closed this task as Resolved.Oct 29 2024, 8:17 AM

Mentioned in SAL (#wikimedia-operations) [2024-10-29T08:24:57Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: T378320', diff saved to https://phabricator.wikimedia.org/P70600 and previous config saved to /var/cache/conftool/dbconfig/20241029-082456-arnaudb.json

Maintenance_bot moved this task from In progress to Done on the DBA board.Oct 29 2024, 8:29 AM

Mentioned in SAL (#wikimedia-operations) [2024-10-29T08:40:06Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1169 (re)pooling @ 75%: T378320', diff saved to https://phabricator.wikimedia.org/P70603 and previous config saved to /var/cache/conftool/dbconfig/20241029-084002-arnaudb.json

Mentioned in SAL (#wikimedia-operations) [2024-10-29T08:55:08Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: T378320', diff saved to https://phabricator.wikimedia.org/P70604 and previous config saved to /var/cache/conftool/dbconfig/20241029-085507-arnaudb.json

db1169 replication broken - enwiki.pagelinks corruptionClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

db1169 replication broken - enwiki.pagelinks corruption
Closed, ResolvedPublic
Actions