MariaDB missing logrotate for error and slow logs
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	Volans
	Feb 21 2016, 10:34 AM

Description

In the DBs configurations we don't set the log_error and slow_query_log_file variables so the default values are used and the files are saved in the data directory and we don't configure a custom logrotate.

Although the slow_query_log is OFF by default and the error log is usually small, in some servers we have it quite big, particularly on masters:

-rw-r----- 1 mysql root    12G Feb 21 09:53 db1052.err
-rw-r----- 1 mysql root    27G Feb 21 10:11 db1038.err

I think we should configure some logrotation policy.

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+4 -0	mariadb: Moved error logs to syslog

Customize query in gerrit

Related Objects

Mentioned In: T120122: Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic)
T127638: Email spam from some MariaDB's logrotate
Mentioned Here: T127638: Email spam from some MariaDB's logrotate

Event Timeline

Volans created this task.Feb 21 2016, 10:34 AM

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 21 2016, 10:34 AM

Volans mentioned this in T127638: Email spam from some MariaDB's logrotate.Feb 21 2016, 10:38 AM

We should send those to syslog/ELK.

Change 272639 had a related patch set uploaded (by Volans):
mariadb: Moved error logs to syslog

https://gerrit.wikimedia.org/r/272639

gerritbot added a project: Patch-For-Review.Feb 23 2016, 12:05 AM

Change 272639 abandoned by Volans:
mariadb: Moved error logs to syslog

https://gerrit.wikimedia.org/r/272639

Complete list of larger error files on the fleet:

db1038.eqiad.wmnet:         33G
dbstore2002.codfw.wmnet:    19G
db1052.eqiad.wmnet:         13G
dbstore2001.codfw.wmnet:    3.7G
db1051.eqiad.wmnet:         2.3G
labsdb1001.eqiad.wmnet:     1.5G
db1011.eqiad.wmnet:         1.3G
db1028.eqiad.wmnet:         839M
db1033.eqiad.wmnet:         626M
db1058.eqiad.wmnet:         386M
db1047.eqiad.wmnet:         364M
db2029.codfw.wmnet:         359M
dbstore1001.eqiad.wmnet:    129M
dbstore1002.eqiad.wmnet:    104M

Volans mentioned this in T120122: Perform a rolling restart of all MySQL slaves (masters too for those services with low traffic).Apr 19 2016, 8:46 PM

Some of the above have been reimaged already/there are blocking tasks to do to reduce its verbosity in the first place.

As I mention on T127638 I would like to work towards ELK if possible.

• 1978Gage2001 moved this task from Triage to In progress on the DBA board.Dec 11 2017, 9:46 AM

Marostegui moved this task from In progress to Triage on the DBA board.Dec 11 2017, 10:56 AM

After my chat with @Volans this is no longer needed as with the migration to stretch logs are no longer stored on /srv/ as they are being handled by journald

MariaDB missing logrotate for error and slow logsClosed, DeclinedPublicActions

Description

Details

Related Objects

Event Timeline

MariaDB missing logrotate for error and slow logs
Closed, DeclinedPublic
Actions