All core hosts (es, x1, sX) should have the event_scheduler = ON and should at least have the events (software/dbtools/events_coredb_slave.sql software/dbtools/events_coredb_master.sql) installed and enabled.
Let's create a non paging alert for those two things.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| In Progress | None | T384810 MariaDB lifetime management system | |||
| Open | None | T376370 mariadb: create a synthetic monitoring indicator for dc switchover readiness | |||
| Resolved | Marostegui | T254738 Create a monitoring check for event_scheduler and for installed events |
Change 604379 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] check_mariadb.py: Add check for the event_scheduler
Change 604379 merged by Marostegui:
[operations/puppet@production] check_mariadb.py: Add check for the event_scheduler
db2089:3315 happened to have all the events disabled and a query running for more than 24h.
root@db2089.codfw.wmnet[ops]> show events; +-----+--------------------------+----------------+-----------+-----------+------------+----------------+----------------+---------------------+------+--------------------+------------+----------------------+----------------------+--------------------+ | Db | Name | Definer | Time zone | Type | Execute at | Interval value | Interval field | Starts | Ends | Status | Originator | character_set_client | collation_connection | Database Collation | +-----+--------------------------+----------------+-----------+-----------+------------+----------------+----------------+---------------------+------+--------------------+------------+----------------------+----------------------+--------------------+ | ops | wmf_slave_overload | root@localhost | SYSTEM | RECURRING | NULL | 10 | SECOND | 2017-04-28 00:00:01 | NULL | SLAVESIDE_DISABLED | 180363369 | utf8 | utf8_general_ci | binary | | ops | wmf_slave_purge | root@localhost | SYSTEM | RECURRING | NULL | 15 | MINUTE | 2017-04-28 00:00:00 | NULL | SLAVESIDE_DISABLED | 180363369 | utf8 | utf8_general_ci | binary | | ops | wmf_slave_wikiuser_sleep | root@localhost | SYSTEM | RECURRING | NULL | 30 | SECOND | 2017-04-28 00:00:05 | NULL | SLAVESIDE_DISABLED | 180363369 | utf8 | utf8_general_ci | binary | | ops | wmf_slave_wikiuser_slow | root@localhost | SYSTEM | RECURRING | NULL | 30 | SECOND | 2017-04-28 00:00:03 | NULL | SLAVESIDE_DISABLED | 180363369 | utf8 | utf8_general_ci | binary | +-----+--------------------------+----------------+-----------+-----------+------------+----------------+----------------+---------------------+------+--------------------+------------+----------------------+----------------------+--------------------+ 4 rows in set (0.033 sec)
I have enabled them back.
Double checked all the production hosts, the above one was the only one without them enabled.
Change 784583 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] monitor_eventscheduler.pp: Add new monitoring check
Change 784583 merged by Marostegui:
[operations/puppet@production] monitor_eventscheduler.pp: Monitor event_scheduler on tests hosts
Change 789037 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] parsercache.pp: Monitor event_scheduler on parsercache
Change 789037 merged by Marostegui:
[operations/puppet@production] parsercache.pp: Monitor event_scheduler on parsercache
Change #1239919 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] monitor_eventscheduler.pp: Monitor event_scheduler on core hosts
Change #1239919 merged by Marostegui:
[operations/puppet@production] core.pp: Monitor event_scheduler on core hosts
Change #1239969 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] mariadb: Add events checker
Change #1239969 merged by Marostegui:
[operations/puppet@production] mariadb: Add events checker
Change #1240286 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] check_mariadb_events.sh: Fixes
Change #1240286 merged by Marostegui:
[operations/puppet@production] check_mariadb_events.sh: Fixes
Tested on db1176 and both worked fine, and sent IRC alerts:
[06:29:47] <+icinga-wm> PROBLEM - MariaDB Event Scheduler test-s4 on db1176 is CRITICAL: CRIT: event_scheduler: False, expected True: OK: Version 10.11.16-MariaDB-log, Uptime 1099299s, read_only: True, 26.73 QPS, connection latency: 0.025371s, query latency: 0.000548s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [06:29:54] <marostegui> ^ me testing [06:30:48] <+icinga-wm> RECOVERY - MariaDB Event Scheduler test-s4 on db1176 is OK: Version 10.11.16-MariaDB-log, Uptime 1099359s, read_only: True, event_scheduler: True, 24.75 QPS, connection latency: 0.029218s, query latency: 0.000491s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler
And:
[06:34:28] <+icinga-wm> PROBLEM - MariaDB Events test-s4 on db1176 is CRITICAL: CRITICAL - Events not ENABLED: wmf_slave_purge(DISABLED) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [06:34:40] <marostegui> ^me testing again [06:35:28] <+icinga-wm> RECOVERY - MariaDB Events test-s4 on db1176 is OK: OK - All 4 events in ops database are ENABLED https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler
Change #1240462 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] core.pp: Add alert for query killers
I've written the wikipage for this alert: https://wikitech.wikimedia.org/w/index.php?title=MariaDB%2FTroubleshooting&diff=2382550&oldid=2361001
Change #1240462 merged by Marostegui:
[operations/puppet@production] core.pp: Add alert for query killers
After testing on core_test.pp hosts I've deployed this to the rest of hosts using query killers and event scheduler, and tested it on a master and a replica.
So right now we have two checks (they do not page, they just notify on IRC)