The logstash for beta does not have any type:syslog messages from deployment-db1 or deployment-db2 https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/syslog
We need central collections of MySQL logs to help investigate side effects of turning on MariaDB strict mode (T108255).
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| beta: send MariaDB errors to syslog | operations/puppet | production | +4 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | None | T108255 Enable MariaDB/MySQL's Strict Mode | |||
| Resolved | hashar | T119370 Send deployment-db1 deployment-db2 syslog to beta cluster logstash | |||
| Declined | None | T140751 Create a logstash input filter to preprocess mysqld syslog messages |
The configuration is around:
deployment-db1:~$ cat /etc/rsyslog.d/30-remote-syslog.conf *.info;mail.none;authpriv.none;cron.none @deployment-logstash2.deployment-prep.eqiad.wmflabs:10514 deployment-db2:~$ cat /etc/rsyslog.d/30-remote-syslog.conf *.info;mail.none;authpriv.none;cron.none @deployment-logstash2.deployment-prep.eqiad.wmflabs:10514
wfLogDBError is already sent to logstash beta, which comes with stracktraces. So syslog is a nice to have but not a priority.
Might MySQL not sending any log at all. On deployment-db1 it runs as:
/usr/sbin/mysqld \ --basedir=/usr --datadir=/mnt/sqldata --plugin-dir=/usr/lib/mysql/plugin \ --user=mysql --log-error=/mnt/sqldata/deployment-db1.err \ --pid-file=/mnt/sqldata/deployment-db1.pid \ --socket=/tmp/mysql.sock --port=3306
/etc/mysql/conf.d has a couple files:
mariadb.cnf
Essentially empty?
# MariaDB-specific config file. # Read by /etc/mysql/my.cnf [client] # Default is Latin1, if you need UTF-8 set this (also in server section) #default-character-set = utf8 [mysqld] # # * Character sets # # Default is Latin1, if you need UTF-8 set all this (also in client section) # #character-set-server = utf8 #collation-server = utf8_general_ci #character_set_server = utf8 #collation_server = utf8_general_ci
mysqld_safe_syslog.cnf
Which references syslog but might not be used:
[mysqld_safe] syslog
There is no /etc/mysql/my.cnf.
The db1 instance has puppet class role::mariadb::beta which invokes:
class { 'mariadb::config': prompt => 'BETA', config => 'mariadb/beta.my.cnf.erb', password => $passwords::misc::scripts::mysql_beta_root_pass, datadir => '/mnt/sqldata', tmpdir => '/mnt/tmp', }
There is no /etc/mysql/my.cnf.
Turns out it is really /etc/my.cnf generated by above mariadb::config class. The my.cnf lacks a log_error = syslog which would presumably causes it to send errors to syslog.
I have no idea how to express that properly in puppet. Maybe hack the ./templates/mariadb/beta.my.cnf.erb file?
./templates/mariadb/beta.my.cnf.erb
yes, if it is not in use by any other server, just add it there.
Change 296713 had a related patch set uploaded (by Hashar):
beta: send MariaDB errors to syslog
Mentioned in SAL [2016-07-06T20:08:04Z] <hashar> beta: on db1 and db2 move the MariaDB 'syslog' setting under [mysqld_safe] section. Cherry picked https://gerrit.wikimedia.org/r/#/c/296713/3 and reloaded mysql on both instances. T119370
Mentioned in SAL [2016-07-06T20:30:50Z] <hashar> beta: restarted mysql on both db1 and db2 so it takes in account the --syslog setting T119370
Both instances show entries for mysql in their /var/log/syslog. The beta cluster logstash is unavailable right now, so I can't really check whether messages are properly relayed and processed by logstash.
Confirmed that both db1 and db2 MySQL send syslog. Messages ends up in Logstash and I copy pasted the wfDBErrror dashboard to a mysql one: https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/mysql
A few messages are aggregated on the emitter side though:
<27>2016-07-14T11:20:23.237747+00:00 mysqld: last message repeated 3 times
That is Precise rsyslogd package coming with the configuration setting RepeatedMsgReduction on https://bugs.launchpad.net/bugs/453444
Thank @jcrespo for the reviews!
Kibana has been upgraded on the beta cluster logstash. I have remade the mysql dashboard. It search for syslog messages matching the program mysqld
https://logstash-beta.wmflabs.org/app/kibana#/dashboard/mysql
Message deduplication does not work quite well since MySQL prefix the message payload with the PID and time. Filled as T140751: Create a logstash input filter to preprocess mysqld syslog messages.