Page MenuHomePhabricator

Email spam from some MariaDB's logrotate
Closed, ResolvedPublic

Description

In our DBs where there is still the mariadb-server-5.5 package or it was removed without the purge option, we still have /etc/logrotate.d/mysql-server that runs every day.

A bunch of things are wrong here:

  • Because the default logrotate from the package don't have the option notifempty, it tries to rotate the empty logs in /var/log
  • Then it fails to run the flush-logs command exiting with EXIT_STATUS=1 hence the email spam.

The reasons why it can't run the flush-logs command are:

  • Wrong socket path in /etc/mysql/debian.cnf: /var/run/mysqld/mysqld.sock instead of /tmp/mysql.sock
  • Missing grant for 'debian-sys-maint'@'localhost'

To avoid the spam the quickest solution is to ensure that the /etc/logrotate.d/mysql-server file is not present from puppet given that is not useful in any case and present only in some server and then decide our custom logrotation in T127636

Event Timeline

Volans created this task.Feb 21 2016, 10:38 AM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 21 2016, 10:38 AM

We do not want mysql-level (flush logs) rotation of logs, as it can cause stalls. We do not want the os or puppet even accessing mysql.

There are some servers still using the 5.5 package; on all others, that log rotate must be deleted, please compile a list.

Technically, most of these will disappear when we deprecate 5.5 on the cluster. I would not do much until then, because we will reimage the old servers.

fgiunchedi triaged this task as Medium priority.Apr 27 2016, 2:30 PM
jcrespo closed this task as Resolved.Jun 8 2016, 10:47 AM
jcrespo claimed this task.

I have deleted all cron jobs on db* hosts trying to rotate mysql logs (we do not allow debian to access mysql in production).

We can create later a puppetization of a proper rotation policy on a separate ticket, but I would prefer to, when working on this, send it directly to ELK.

The above tickets exists and it is: T127636