Page MenuHomePhabricator

Create a logstash input filter to preprocess mysqld syslog messages
Open, MediumPublic

Description

On the beta cluster, we had mysqld to emit syslog which are relayed to logstash (T119370). I have crafted a basic dashboard that search for syslog messages matching the program mysqld:

https://logstash-beta.wmflabs.org/app/kibana#/dashboard/mysql

Message deduplication does not work quite well since MySQL prefix the message payload with the PID and time.

Seems MariaDB emits all messages at error level (havent checked).

They look like (message shortened by me):

_typesyslog
facility3
facility_labelsystem
levelERROR
severity3
severity_labelError
programmysqld
message160719 9:20:36 [Warning] Unsafe statement written to the binary ...
normalized_message160719 9:20:36 [Warning] Unsafe statement written to the binary ...

At least in the message field, we would want to drop the PID and time.

Ideally parse the message to set / correct the level.


From https://mariadb.com/kb/en/mariadb/error-log/#format

Until MariaDB 10.1.4, the format consisted of the date (yymmdd) and time, followed by the type of error (Note, Warning or Error) and the error message, for example:

160615 16:53:08 [Note] InnoDB: The InnoDB memory heap is disabled

From MariaDB 10.1.5, the date format has been extended to yyyy-mm-dd, and the thread ID has been added, for example:

2016-06-15 16:53:33 139651251140544 [Note] InnoDB: The InnoDB memory heap is disabled

The type of error is one of Note, Warning or Error.

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 19 2016, 10:29 AM
jcrespo added a comment.EditedJul 19 2016, 10:37 AM

This is a known issue, and I cannot find one now, but there are numerous examples found elsewere of people that have done that before and shared it.

One thing that you may want for beta, that we could not do for production, is to enable more verbose logging such as the slow log and the general log. It will depend on the resources available and the traffic. But they can be tuned and they would be very useful if debugging is needed (it may be even useful to have them prepared, but not enabled).

hashar updated the task description. (Show Details)Jul 19 2016, 10:40 AM

Be careful because on start/stop, you will find some multi-line "errors" (aside from a proper error log, it also serves as a core dump indication and recovery log).

1978Gage2001 moved this task from Triage to In progress on the DBA board.Dec 11 2017, 9:45 AM
Marostegui moved this task from In progress to Triage on the DBA board.Dec 11 2017, 10:57 AM
fgiunchedi moved this task from Inbox to Radar on the observability board.Dec 9 2019, 11:53 AM