Page MenuHomePhabricator

Send deployment-db1 deployment-db2 syslog to beta cluster logstash
Closed, ResolvedPublic


The logstash for beta does not have any type:syslog messages from deployment-db1 or deployment-db2

We need central collections of MySQL logs to help investigate side effects of turning on MariaDB strict mode (T108255).

Event Timeline

hashar raised the priority of this task from to Medium.
hashar updated the task description. (Show Details)
hashar added subscribers: zeljkofilipin, hashar, wpmirrordev and 14 others.

The configuration is around:

deployment-db1:~$ cat /etc/rsyslog.d/30-remote-syslog.conf 
*.info;mail.none;authpriv.none;cron.none @deployment-logstash2.deployment-prep.eqiad.wmflabs:10514

deployment-db2:~$ cat /etc/rsyslog.d/30-remote-syslog.conf 
*.info;mail.none;authpriv.none;cron.none @deployment-logstash2.deployment-prep.eqiad.wmflabs:10514
hashar lowered the priority of this task from Medium to Low.Nov 23 2015, 10:49 AM

wfLogDBError is already sent to logstash beta, which comes with stracktraces. So syslog is a nice to have but not a priority.

Might MySQL not sending any log at all. On deployment-db1 it runs as:

/usr/sbin/mysqld \
 --basedir=/usr --datadir=/mnt/sqldata --plugin-dir=/usr/lib/mysql/plugin \
 --user=mysql --log-error=/mnt/sqldata/deployment-db1.err \
 --pid-file=/mnt/sqldata/ \
 --socket=/tmp/mysql.sock --port=3306

/etc/mysql/conf.d has a couple files:

Essentially empty?

# MariaDB-specific config file.
# Read by /etc/mysql/my.cnf

# Default is Latin1, if you need UTF-8 set this (also in server section)
#default-character-set = utf8 

# * Character sets
# Default is Latin1, if you need UTF-8 set all this (also in client section)
#character-set-server  = utf8 
#collation-server      = utf8_general_ci 
#character_set_server   = utf8 
#collation_server       = utf8_general_ci 


Which references syslog but might not be used:


There is no /etc/mysql/my.cnf.

The db1 instance has puppet class role::mariadb::beta which invokes:

class { 'mariadb::config':
    prompt   => 'BETA',
    config   => 'mariadb/',
    password => $passwords::misc::scripts::mysql_beta_root_pass,
    datadir  => '/mnt/sqldata',
    tmpdir   => '/mnt/tmp',

There is no /etc/mysql/my.cnf.

Turns out it is really /etc/my.cnf generated by above mariadb::config class. The my.cnf lacks a log_error = syslog which would presumably causes it to send errors to syslog.

I have no idea how to express that properly in puppet. Maybe hack the ./templates/mariadb/ file?


yes, if it is not in use by any other server, just add it there.

Change 296713 had a related patch set uploaded (by Hashar):
beta: send MariaDB errors to syslog

Mentioned in SAL [2016-07-06T20:08:04Z] <hashar> beta: on db1 and db2 move the MariaDB 'syslog' setting under [mysqld_safe] section. Cherry picked and reloaded mysql on both instances. T119370

Mentioned in SAL [2016-07-06T20:30:50Z] <hashar> beta: restarted mysql on both db1 and db2 so it takes in account the --syslog setting T119370

Both instances show entries for mysql in their /var/log/syslog. The beta cluster logstash is unavailable right now, so I can't really check whether messages are properly relayed and processed by logstash.

Change 296713 merged by Jcrespo:
beta: send MariaDB errors to syslog

Confirmed that both db1 and db2 MySQL send syslog. Messages ends up in Logstash and I copy pasted the wfDBErrror dashboard to a mysql one:

A few messages are aggregated on the emitter side though:

<27>2016-07-14T11:20:23.237747+00:00 mysqld: last message repeated 3 times

That is Precise rsyslogd package coming with the configuration setting RepeatedMsgReduction on

Thank @jcrespo for the reviews!

Kibana has been upgraded on the beta cluster logstash. I have remade the mysql dashboard. It search for syslog messages matching the program mysqld

Message deduplication does not work quite well since MySQL prefix the message payload with the PID and time. Filled as T140751: Create a logstash input filter to preprocess mysqld syslog messages.

This comment was removed by jcrespo.