Page MenuHomePhabricator
Create Task
Maniphest T359597

db2124 depooled with index corruption
Closed, ResolvedPublic
Actions

Description

Hi DBA,

22:07:45 <+icinga-wm> PROBLEM - MariaDB Replica SQL: s6 #page on db2124 is CRITICAL: CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1034, Errmsg: Error Index for table page_props is corrupt: try to repair it on query. Default database: frwiki. [Query snipped] https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica

I've depooled it at 22:10, and I'm about to downtime (for five days, through this time Tuesday, just in case SRE Summit travel means it takes longer to look at).

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
db2124: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects

Event Timeline

RLazarus created this task.Mar 7 2024, 10:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 7 2024, 10:15 PM
ops-monitoring-bot added a comment.Mar 7 2024, 10:17 PM

Icinga downtime and Alertmanager silence (ID=17885a36-8547-4a13-afea-8d73c87e272d) set by rzl@cumin2002 for 5 days, 0:00:00 on 1 host(s) and their services with reason: index corruption

db2124.codfw.wmnet
RLazarus triaged this task as High priority.Mar 7 2024, 10:21 PM
RLazarus added a comment.Mar 7 2024, 10:33 PM

I don't see anything obviously hardware-broken in logs. I notice it was just repooled yesterday after maintenance for T352010, but nothing jumps out as an obvious cause. Over to the DBAs from here, enjoy. :)

Marostegui claimed this task.Mar 8 2024, 6:26 AM
Marostegui subscribed.

I could probably fix it right away, but I think I am going to quickly reclone it instead

Marostegui moved this task from Triage to In progress on the DBA board.Mar 8 2024, 6:30 AM
gerritbot added a comment.Mar 8 2024, 6:31 AM

Change 1009638 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2124: Disable notifications

https://gerrit.wikimedia.org/r/1009638

gerritbot added a project: Patch-For-Review.Mar 8 2024, 6:31 AM

Change 1009638 merged by Marostegui:

[operations/puppet@production] db2124: Disable notifications

https://gerrit.wikimedia.org/r/1009638

Marostegui closed this task as Resolved.Mar 8 2024, 7:20 AM

Host recloned and being slowly repooled.
Thanks @RLazarus for addressing this incident!

Maintenance_bot moved this task from In progress to Done on the DBA board.Mar 8 2024, 7:29 AM
Maintenance_bot removed a project: Patch-For-Review.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits