Page MenuHomePhabricator

MariaDB missing logrotate for error and slow logs
Closed, DeclinedPublic

Description

In the DBs configurations we don't set the log_error and slow_query_log_file variables so the default values are used and the files are saved in the data directory and we don't configure a custom logrotate.

Although the slow_query_log is OFF by default and the error log is usually small, in some servers we have it quite big, particularly on masters:

-rw-r----- 1 mysql root    12G Feb 21 09:53 db1052.err
-rw-r----- 1 mysql root    27G Feb 21 10:11 db1038.err

I think we should configure some logrotation policy.

Details

Related Gerrit Patches:
operations/puppet : productionmariadb: Moved error logs to syslog

Event Timeline

Volans created this task.Feb 21 2016, 10:34 AM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 21 2016, 10:34 AM

We should send those to syslog/ELK.

Change 272639 had a related patch set uploaded (by Volans):
mariadb: Moved error logs to syslog

https://gerrit.wikimedia.org/r/272639

Change 272639 abandoned by Volans:
mariadb: Moved error logs to syslog

https://gerrit.wikimedia.org/r/272639

Complete list of larger error files on the fleet:

db1038.eqiad.wmnet:         33G
dbstore2002.codfw.wmnet:    19G
db1052.eqiad.wmnet:         13G
dbstore2001.codfw.wmnet:    3.7G
db1051.eqiad.wmnet:         2.3G
labsdb1001.eqiad.wmnet:     1.5G
db1011.eqiad.wmnet:         1.3G
db1028.eqiad.wmnet:         839M
db1033.eqiad.wmnet:         626M
db1058.eqiad.wmnet:         386M
db1047.eqiad.wmnet:         364M
db2029.codfw.wmnet:         359M
dbstore1001.eqiad.wmnet:    129M
dbstore1002.eqiad.wmnet:    104M

Some of the above have been reimaged already/there are blocking tasks to do to reduce its verbosity in the first place.

As I mention on T127638 I would like to work towards ELK if possible.

1978Gage2001 moved this task from Triage to In progress on the DBA board.Dec 11 2017, 9:46 AM
Marostegui moved this task from In progress to Triage on the DBA board.Dec 11 2017, 10:56 AM
Marostegui closed this task as Declined.Jun 11 2018, 8:12 AM
Marostegui added a subscriber: Marostegui.

After my chat with @Volans this is no longer needed as with the migration to stretch logs are no longer stored on /srv/ as they are being handled by journald