Page MenuHomePhabricator

logrotate and logstash logs does not play well together
Open, Needs TriagePublic

Description

Rotation of logs seems to be handled by logstash internal loggers, when logrotate is applied it generates useless files like :

-rw-r--r-- 1 logstash logstash     260912 May 10 06:25 /var/log/logstash/logstash-plain-2017-05-08.log.1
-rw-r--r-- 1 logstash logstash       3906 May 11 06:25 /var/log/logstash/logstash-plain-2017-05-09.log.1
-rw-r--r-- 1 logstash logstash      19510 May 12 06:25 /var/log/logstash/logstash-plain-2017-05-10.log.1
-rw-r--r-- 1 logstash logstash          0 May 13 06:25 /var/log/logstash/logstash-plain-2017-05-11.log
-rw-r--r-- 1 logstash logstash 3482931190 May 13 06:25 /var/log/logstash/logstash-plain-2017-05-11.log.1
-rw-r--r-- 1 logstash logstash          0 May 14 06:25 /var/log/logstash/logstash-plain-2017-05-12.log
-rw-r--r-- 1 logstash logstash      87494 May 14 06:25 /var/log/logstash/logstash-plain-2017-05-12.log.1
-rw-r--r-- 1 logstash logstash          0 May 15 06:25 /var/log/logstash/logstash-plain-2017-05-13.log
-rw-r--r-- 1 logstash logstash     208525 May 15 06:25 /var/log/logstash/logstash-plain-2017-05-13.log.1
-rw-r--r-- 1 logstash logstash          0 May 16 06:25 /var/log/logstash/logstash-plain-2017-05-14.log
-rw-r--r-- 1 logstash logstash     242044 May 16 06:25 /var/log/logstash/logstash-plain-2017-05-14.log.1
-rw-r--r-- 1 logstash logstash          0 May 17 06:25 /var/log/logstash/logstash-plain-2017-05-15.log
-rw-r--r-- 1 logstash logstash      88680 May 17 06:25 /var/log/logstash/logstash-plain-2017-05-15.log.1
-rw-r--r-- 1 logstash logstash          0 May 18 06:25 /var/log/logstash/logstash-plain-2017-05-16.log
-rw-r--r-- 1 logstash logstash     364263 May 18 06:25 /var/log/logstash/logstash-plain-2017-05-16.log.1
-rw-r--r-- 1 logstash logstash          0 May 19 06:25 /var/log/logstash/logstash-plain-2017-05-17.log
-rw-r--r-- 1 logstash logstash    1348902 May 19 06:25 /var/log/logstash/logstash-plain-2017-05-17.log.1
-rw-r--r-- 1 logstash logstash          0 May 20 06:25 /var/log/logstash/logstash-plain-2017-05-18.log
-rw-r--r-- 1 logstash logstash     138201 May 20 06:25 /var/log/logstash/logstash-plain-2017-05-18.log.1
-rw-r--r-- 1 logstash logstash          0 May 21 06:25 /var/log/logstash/logstash-plain-2017-05-19.log
-rw-r--r-- 1 logstash logstash     104694 May 21 06:25 /var/log/logstash/logstash-plain-2017-05-19.log.1
-rw-r--r-- 1 logstash logstash          0 May 22 06:25 /var/log/logstash/logstash-plain-2017-05-20.log
-rw-r--r-- 1 logstash logstash     143033 May 22 06:25 /var/log/logstash/logstash-plain-2017-05-20.log.1
-rw-r--r-- 1 logstash logstash          0 May 23 06:25 /var/log/logstash/logstash-plain-2017-05-21.log
-rw-r--r-- 1 logstash logstash     118422 May 23 06:25 /var/log/logstash/logstash-plain-2017-05-21.log.1
-rw-r--r-- 1 logstash logstash     609769 May 22 23:58 /var/log/logstash/logstash-plain-2017-05-22.log

Event Timeline

dcausse created this task.May 23 2017, 8:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 23 2017, 8:20 AM

AFAICS we're not using logrotate anymore for /var/log/logstash, but we're not cleaning up old logs either:

logstash1008:~$ ls -latr /var/log/logstash/*plain* | head -10
-rw-r--r-- 1 logstash logstash     113235 Feb 20 23:59 /var/log/logstash/logstash-plain-2019-02-20.log
-rw-r--r-- 1 logstash logstash    1945637 Feb 21 23:59 /var/log/logstash/logstash-plain-2019-02-21.log
-rw-r--r-- 1 logstash logstash    6352743 Feb 22 23:59 /var/log/logstash/logstash-plain-2019-02-22.log
-rw-r--r-- 1 logstash logstash    1906936 Feb 23 23:59 /var/log/logstash/logstash-plain-2019-02-23.log
-rw-r--r-- 1 logstash logstash    4223296 Feb 24 23:59 /var/log/logstash/logstash-plain-2019-02-24.log
-rw-r--r-- 1 logstash logstash    5054570 Feb 25 23:59 /var/log/logstash/logstash-plain-2019-02-25.log
-rw-r--r-- 1 logstash logstash   11167466 Feb 26 23:59 /var/log/logstash/logstash-plain-2019-02-26.log
-rw-r--r-- 1 logstash logstash    8048046 Feb 27 23:59 /var/log/logstash/logstash-plain-2019-02-27.log
-rw-r--r-- 1 logstash logstash    9035515 Feb 28 23:59 /var/log/logstash/logstash-plain-2019-02-28.log
-rw-r--r-- 1 logstash logstash     636535 Mar  1 23:59 /var/log/logstash/logstash-plain-2019-03-01.log

For logstash logs themselves I think we should ask log4j2 to cleanup, e.g. https://www.elastic.co/guide/en/elasticsearch/reference/6.5/logging.html

For gc logs I think we can use logrotate, I don't think the jvm is able to cleanup old files by itself?

A third option is to use systemd-tmpfiles which supports cleaning up directories / file globs

Change 528305 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: manage currently unmanaged log4j2.properties file

https://gerrit.wikimedia.org/r/528305

Change 528306 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: rotate logstash plain logs with log4j2

https://gerrit.wikimedia.org/r/528306

Change 528305 merged by Herron:
[operations/puppet@production] logstash: manage currently unmanaged log4j2.properties file

https://gerrit.wikimedia.org/r/528305

Mentioned in SAL (#wikimedia-operations) [2019-08-06T19:50:39Z] <herron> disabling puppet on logstash collectors for rolling deploy of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528306/ T166107

Change 528306 merged by Herron:
[operations/puppet@production] logstash: rotate logstash plain logs with log4j2

https://gerrit.wikimedia.org/r/528306