Page MenuHomePhabricator

Update DB read_only alert to represent correct state
Open, MediumPublic

Description

As an example, the current 'read_only' alert will fire both for masters (where the expected result is 'false') and replicas (where the expected result is 'true'). In both cases it points to docs that are only relevant to one specific case: https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only

It would be better to have a different alert for master vs replica, and to point to relevant docs for both.

Event Timeline

LSobanski renamed this task from Make alerts more specific to Make DB alerts more specific.Mar 11 2021, 2:01 PM

Sample IRC alert:

11:37:32 <+icinga-wm> PROBLEM - MariaDB read only pc1 #page on pc1010 is CRITICAL: CRIT: read_only: False, expected True: OK: Version 10.4.18-MariaDB-log, Uptime 873607s, event_scheduler: True, 1098.38 QPS, connection latency: 0.003498s, query latency: 0.000608s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only
LSobanski triaged this task as Medium priority.Mar 11 2021, 2:05 PM
LSobanski moved this task from Triage to Refine on the DBA board.
LSobanski renamed this task from Make DB alerts more specific to Update DB read_only alert to represent correct state.Mar 15 2021, 7:07 PM