Page MenuHomePhabricator

Cleanup old logstash logs (application and JVM GC)
Open, MediumPublic

Description

Rotation of logs seems to be handled by logstash internal loggers, when logrotate is applied it generates useless files like :

-rw-r--r-- 1 logstash logstash     260912 May 10 06:25 /var/log/logstash/logstash-plain-2017-05-08.log.1
-rw-r--r-- 1 logstash logstash       3906 May 11 06:25 /var/log/logstash/logstash-plain-2017-05-09.log.1
-rw-r--r-- 1 logstash logstash      19510 May 12 06:25 /var/log/logstash/logstash-plain-2017-05-10.log.1
-rw-r--r-- 1 logstash logstash          0 May 13 06:25 /var/log/logstash/logstash-plain-2017-05-11.log
-rw-r--r-- 1 logstash logstash 3482931190 May 13 06:25 /var/log/logstash/logstash-plain-2017-05-11.log.1
-rw-r--r-- 1 logstash logstash          0 May 14 06:25 /var/log/logstash/logstash-plain-2017-05-12.log
-rw-r--r-- 1 logstash logstash      87494 May 14 06:25 /var/log/logstash/logstash-plain-2017-05-12.log.1
-rw-r--r-- 1 logstash logstash          0 May 15 06:25 /var/log/logstash/logstash-plain-2017-05-13.log
-rw-r--r-- 1 logstash logstash     208525 May 15 06:25 /var/log/logstash/logstash-plain-2017-05-13.log.1
-rw-r--r-- 1 logstash logstash          0 May 16 06:25 /var/log/logstash/logstash-plain-2017-05-14.log
-rw-r--r-- 1 logstash logstash     242044 May 16 06:25 /var/log/logstash/logstash-plain-2017-05-14.log.1
-rw-r--r-- 1 logstash logstash          0 May 17 06:25 /var/log/logstash/logstash-plain-2017-05-15.log
-rw-r--r-- 1 logstash logstash      88680 May 17 06:25 /var/log/logstash/logstash-plain-2017-05-15.log.1
-rw-r--r-- 1 logstash logstash          0 May 18 06:25 /var/log/logstash/logstash-plain-2017-05-16.log
-rw-r--r-- 1 logstash logstash     364263 May 18 06:25 /var/log/logstash/logstash-plain-2017-05-16.log.1
-rw-r--r-- 1 logstash logstash          0 May 19 06:25 /var/log/logstash/logstash-plain-2017-05-17.log
-rw-r--r-- 1 logstash logstash    1348902 May 19 06:25 /var/log/logstash/logstash-plain-2017-05-17.log.1
-rw-r--r-- 1 logstash logstash          0 May 20 06:25 /var/log/logstash/logstash-plain-2017-05-18.log
-rw-r--r-- 1 logstash logstash     138201 May 20 06:25 /var/log/logstash/logstash-plain-2017-05-18.log.1
-rw-r--r-- 1 logstash logstash          0 May 21 06:25 /var/log/logstash/logstash-plain-2017-05-19.log
-rw-r--r-- 1 logstash logstash     104694 May 21 06:25 /var/log/logstash/logstash-plain-2017-05-19.log.1
-rw-r--r-- 1 logstash logstash          0 May 22 06:25 /var/log/logstash/logstash-plain-2017-05-20.log
-rw-r--r-- 1 logstash logstash     143033 May 22 06:25 /var/log/logstash/logstash-plain-2017-05-20.log.1
-rw-r--r-- 1 logstash logstash          0 May 23 06:25 /var/log/logstash/logstash-plain-2017-05-21.log
-rw-r--r-- 1 logstash logstash     118422 May 23 06:25 /var/log/logstash/logstash-plain-2017-05-21.log.1
-rw-r--r-- 1 logstash logstash     609769 May 22 23:58 /var/log/logstash/logstash-plain-2017-05-22.log

And GC logs are left behind too:

logstash1007:/var/log/logstash$ ls -latr *gc*
-rw-r--r-- 1 logstash logstash 20972287 Feb 24  2019 logstash_jvm_gc.pid31414.log.4
-rw-r--r-- 1 logstash logstash 20972207 Feb 25  2019 logstash_jvm_gc.pid31414.log.5
-rw-r--r-- 1 logstash logstash 20972049 Feb 25  2019 logstash_jvm_gc.pid31414.log.6
-rw-r--r-- 1 logstash logstash 20972190 Feb 26  2019 logstash_jvm_gc.pid31414.log.7
-rw-r--r-- 1 logstash logstash 20971735 Feb 26  2019 logstash_jvm_gc.pid31414.log.8
-rw-r--r-- 1 logstash logstash 20971894 Feb 27  2019 logstash_jvm_gc.pid31414.log.9
-rw-r--r-- 1 logstash logstash 20971879 Feb 27  2019 logstash_jvm_gc.pid31414.log.0
-rw-r--r-- 1 logstash logstash 20971736 Feb 27  2019 logstash_jvm_gc.pid31414.log.1
-rw-r--r-- 1 logstash logstash 20971783 Feb 28  2019 logstash_jvm_gc.pid31414.log.2
-rw-r--r-- 1 logstash logstash  9669845 Feb 28  2019 logstash_jvm_gc.pid31414.log.3.current
-rw-r--r-- 1 logstash logstash  6012996 Feb 28  2019 logstash_jvm_gc.pid1266.log.0.current
-rw-r--r-- 1 logstash logstash  4015898 Feb 28  2019 logstash_jvm_gc.pid18863.log.0.current
-rw-r--r-- 1 logstash logstash   211578 Feb 28  2019 logstash_jvm_gc.pid3230.log.0.current
-rw-r--r-- 1 logstash logstash   262391 Feb 28  2019 logstash_jvm_gc.pid5708.log.0.current
-rw-r--r-- 1 logstash logstash   176543 Feb 28  2019 logstash_jvm_gc.pid9331.log.0.current
-rw-r--r-- 1 logstash logstash  1016132 Feb 28  2019 logstash_jvm_gc.pid11434.log.0.current
-rw-r--r-- 1 logstash logstash 20972055 Mar  1  2019 logstash_jvm_gc.pid16200.log.0
-rw-r--r-- 1 logstash logstash 20972012 Mar  1  2019 logstash_jvm_gc.pid16200.log.1

Details

Related Gerrit Patches:

Event Timeline

dcausse created this task.May 23 2017, 8:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 23 2017, 8:20 AM

AFAICS we're not using logrotate anymore for /var/log/logstash, but we're not cleaning up old logs either:

logstash1008:~$ ls -latr /var/log/logstash/*plain* | head -10
-rw-r--r-- 1 logstash logstash     113235 Feb 20 23:59 /var/log/logstash/logstash-plain-2019-02-20.log
-rw-r--r-- 1 logstash logstash    1945637 Feb 21 23:59 /var/log/logstash/logstash-plain-2019-02-21.log
-rw-r--r-- 1 logstash logstash    6352743 Feb 22 23:59 /var/log/logstash/logstash-plain-2019-02-22.log
-rw-r--r-- 1 logstash logstash    1906936 Feb 23 23:59 /var/log/logstash/logstash-plain-2019-02-23.log
-rw-r--r-- 1 logstash logstash    4223296 Feb 24 23:59 /var/log/logstash/logstash-plain-2019-02-24.log
-rw-r--r-- 1 logstash logstash    5054570 Feb 25 23:59 /var/log/logstash/logstash-plain-2019-02-25.log
-rw-r--r-- 1 logstash logstash   11167466 Feb 26 23:59 /var/log/logstash/logstash-plain-2019-02-26.log
-rw-r--r-- 1 logstash logstash    8048046 Feb 27 23:59 /var/log/logstash/logstash-plain-2019-02-27.log
-rw-r--r-- 1 logstash logstash    9035515 Feb 28 23:59 /var/log/logstash/logstash-plain-2019-02-28.log
-rw-r--r-- 1 logstash logstash     636535 Mar  1 23:59 /var/log/logstash/logstash-plain-2019-03-01.log

For logstash logs themselves I think we should ask log4j2 to cleanup, e.g. https://www.elastic.co/guide/en/elasticsearch/reference/6.5/logging.html

For gc logs I think we can use logrotate, I don't think the jvm is able to cleanup old files by itself?

A third option is to use systemd-tmpfiles which supports cleaning up directories / file globs

Change 528305 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: manage currently unmanaged log4j2.properties file

https://gerrit.wikimedia.org/r/528305

Change 528306 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: rotate logstash plain logs with log4j2

https://gerrit.wikimedia.org/r/528306

Change 528305 merged by Herron:
[operations/puppet@production] logstash: manage currently unmanaged log4j2.properties file

https://gerrit.wikimedia.org/r/528305

Mentioned in SAL (#wikimedia-operations) [2019-08-06T19:50:39Z] <herron> disabling puppet on logstash collectors for rolling deploy of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528306/ T166107

Change 528306 merged by Herron:
[operations/puppet@production] logstash: rotate logstash plain logs with log4j2

https://gerrit.wikimedia.org/r/528306

The original issue is gone, taking over this issue for general cleanup on logstash logs (including GC)

fgiunchedi renamed this task from logrotate and logstash logs does not play well together to Cleanup old logstash logs (application and JVM GC).Dec 9 2019, 11:50 AM
fgiunchedi triaged this task as Medium priority.
fgiunchedi updated the task description. (Show Details)
fgiunchedi moved this task from Inbox to Up next on the observability board.

It doesn't look like the current log4j config is working as intended:

root@logstash2023:/var/log/logstash# ls -lat *plain* | tail -3
-rw-r--r-- 1 logstash logstash  20344739 Dec  5 00:00 logstash-plain-2019-12-04.log
-rw-r--r-- 1 logstash logstash  35575161 Dec  4 00:00 logstash-plain-2019-12-03.log
-rw-r--r-- 1 logstash logstash   3101717 Dec  3 00:00 logstash-plain-2019-12-02.log

Similarly, the gc logs rotation (openjdk 11 on buster) config doesn't seem to work:

root@logstash2023:/var/log/logstash# ls -lat *gc* | tail -3
-rw-r--r-- 1 logstash logstash 16827 Dec  2 21:14 logstash_jvm_gc.30722.log
-rw-r--r-- 1 logstash logstash 16003 Dec  2 21:14 logstash_jvm_gc.29816.log
-rw-r--r-- 1 logstash logstash 18618 Dec  2 21:13 logstash_jvm_gc.29651.log.0