Page MenuHomePhabricator

Use Gerrit 3.10 built-in log rotation
Closed, ResolvedPublic

Description

Gerrit 3.10 comes with a log.timeToKeep setting to delete old log files. We currently rely on an home made systemd timer:

modules/gerrit/manifests/jobs.pp
class gerrit::jobs {
    systemd::timer::job { 'clear_gerrit_logs':
        ensure      => present,
        user        => 'root',
        description => 'Gerrit rotates their own logs, but does not clean them out. Delete old logs.',
        command     => "/usr/bin/find /var/log/gerrit/ -name \"*.gz\" -mtime +30 -delete",
        interval    => {'start' => 'OnCalendar', 'interval' => 'daily'},
    }
}

Once upgraded, we can drop the timed find in favor of having Gerrit to do the rotation.

Related Objects

Event Timeline

Change #1049090 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: enable built-in log rotation

https://gerrit.wikimedia.org/r/1049090

Change #1049091 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: remove absented clear_gerrit_logs timer job

https://gerrit.wikimedia.org/r/1049091

Change #1049090 merged by Dzahn:

[operations/puppet@production] gerrit: enable built-in log rotation

https://gerrit.wikimedia.org/r/1049090

I have confirmed the systemd timer is gone and no more rotating:

sudo journalctl -u clear_gerrit_logs --since "2 days ago"
-- Journal begins at Fri 2024-06-07 23:22:12 UTC, ends at Tue 2024-07-09 12:52:12 UTC. --
Jul 08 00:00:01 gerrit1003 systemd[1]: Starting Gerrit rotates their own logs, but does not clean them out. Delete old logs....
Jul 08 00:00:01 gerrit1003 systemd[1]: clear_gerrit_logs.service: Succeeded.
Jul 08 00:00:01 gerrit1003 systemd[1]: Finished Gerrit rotates their own logs, but does not clean them out. Delete old logs..

The hosts have:

gerrit1003:~$ ls -tl /var/log/gerrit/|tail -n7
-r--r--r-- 1 gerrit2 gerrit2   917101 Jun  8 23:00 replication_log.2024-06-07.gz
-r--r--r-- 1 gerrit2 gerrit2   204088 Jun  7 23:00 error_log.json.2024-06-06.gz
-r--r--r-- 1 gerrit2 gerrit2  1373241 Jun  7 23:00 replication_log.2024-06-06.gz
-r--r--r-- 1 gerrit2 gerrit2  1458736 Jun  7 23:00 sshd_log.2024-06-06.gz
-r--r--r-- 1 gerrit2 gerrit2   170484 Jun  7 23:00 error_log.2024-06-06.gz
-r--r--r-- 1 gerrit2 gerrit2  1702498 Jun  7 23:00 sshd_log.json.2024-06-06.gz
-rw-r--r-- 1 gerrit2 gerrit2        0 Feb  5 14:45 rename_log
gerrit2002:~$ ls -lt /var/log/gerrit/|tail -n7
-r--r--r-- 1 gerrit2 gerrit2      88 Jun  8 23:00 gc_log.2024-06-01.gz
-r--r--r-- 1 gerrit2 gerrit2    3144 Jun  8 23:00 error_log.json.2024-06-07.gz
-r--r--r-- 1 gerrit2 gerrit2    2003 Jun  7 23:00 error_log.2024-06-06.gz
-r--r--r-- 1 gerrit2 gerrit2    2658 Jun  7 23:00 error_log.json.2024-06-06.gz
-rw-r--r-- 1 gerrit2 gerrit2       0 Feb  5 14:45 rename_log
-rw-r--r-- 1 gerrit2 gerrit2       0 May 25  2023 replication_log
-rw-r--r-- 1 gerrit2 gerrit2       0 May 25  2023 delete_log

Mentioned in SAL (#wikimedia-operations) [2024-07-09T12:59:47Z] <hashar> Restart Gerrit replica on gerrit2002 to apply a configuration change | T367505

And surely after restarting Gerrit on gerrit2002, the logs got cleaned:

ls -lt /var/log/gerrit/|tail -n7
-r--r--r-- 1 gerrit2 gerrit2    1972 Jun 10 08:24 error_log.2024-06-09.gz
-r--r--r-- 1 gerrit2 gerrit2    2685 Jun 10 08:24 error_log.json.2024-06-09.gz
-r--r--r-- 1 gerrit2 gerrit2    1975 Jun  9 23:00 error_log.2024-06-08.gz
-r--r--r-- 1 gerrit2 gerrit2    2695 Jun  9 23:00 error_log.json.2024-06-08.gz
-rw-r--r-- 1 gerrit2 gerrit2       0 Feb  5 14:45 rename_log
-rw-r--r-- 1 gerrit2 gerrit2       0 May 25  2023 replication_log
-rw-r--r-- 1 gerrit2 gerrit2       0 May 25  2023 delete_log

Success!

Mentioned in SAL (#wikimedia-operations) [2024-07-09T14:50:24Z] <hashar> Restart Gerrit primary on gerrit1003 to apply a configuration change | T367505

Same effect on gerrit1003:

gerrit1003:~$ ls -lt /var/log/gerrit/|tail -n7
-r--r--r-- 1 gerrit2 gerrit2   410968 Jun 10 08:29 replication_log.2024-06-09.gz
-r--r--r-- 1 gerrit2 gerrit2   692693 Jun  9 23:00 sshd_log.json.2024-06-08.gz
-r--r--r-- 1 gerrit2 gerrit2   220093 Jun  9 23:00 error_log.2024-06-08.gz
-r--r--r-- 1 gerrit2 gerrit2   252207 Jun  9 23:00 error_log.json.2024-06-08.gz
-r--r--r-- 1 gerrit2 gerrit2   595531 Jun  9 23:00 sshd_log.2024-06-08.gz
-r--r--r-- 1 gerrit2 gerrit2   667359 Jun  9 23:00 replication_log.2024-06-08.gz
-rw-r--r-- 1 gerrit2 gerrit2        0 Feb  5 14:45 rename_log

I will revisit tomorrow to ensure log files got properly deleted.

The Gerrit instance do trigger the log maintenance:

gerrit_log_maintenance.png (329×907 px, 67 KB)

I will check again tomorrow to confirm the old files got deleted.

Same effect on gerrit1003:

gerrit1003:~$ ls -lt /var/log/gerrit/|tail -n7
-r--r--r-- 1 gerrit2 gerrit2   410968 Jun 10 08:29 replication_log.2024-06-09.gz
-r--r--r-- 1 gerrit2 gerrit2   692693 Jun  9 23:00 sshd_log.json.2024-06-08.gz
-r--r--r-- 1 gerrit2 gerrit2   220093 Jun  9 23:00 error_log.2024-06-08.gz
-r--r--r-- 1 gerrit2 gerrit2   252207 Jun  9 23:00 error_log.json.2024-06-08.gz
-r--r--r-- 1 gerrit2 gerrit2   595531 Jun  9 23:00 sshd_log.2024-06-08.gz
-r--r--r-- 1 gerrit2 gerrit2   667359 Jun  9 23:00 replication_log.2024-06-08.gz
-rw-r--r-- 1 gerrit2 gerrit2        0 Feb  5 14:45 rename_log

I will revisit tomorrow to ensure log files got properly deleted.

I can confirm the log got rotated:

gerrit1003$ ls -lt /var/log/gerrit/|tail -n7
-r--r--r-- 1 gerrit2 gerrit2  1390868 Jun 12 23:00 sshd_log.2024-06-11.gz
-r--r--r-- 1 gerrit2 gerrit2  1242215 Jun 11 23:00 sshd_log.2024-06-10.gz
-r--r--r-- 1 gerrit2 gerrit2   434247 Jun 11 23:00 error_log.2024-06-10.gz
-r--r--r-- 1 gerrit2 gerrit2   533065 Jun 11 23:00 error_log.json.2024-06-10.gz
-r--r--r-- 1 gerrit2 gerrit2  1533221 Jun 11 23:00 replication_log.2024-06-10.gz
-r--r--r-- 1 gerrit2 gerrit2  1453863 Jun 11 23:00 sshd_log.json.2024-06-10.gz
-rw-r--r-- 1 gerrit2 gerrit2        0 Feb  5 14:45 rename_log

And

gerrit2002:~$ ls -lt /var/log/gerrit/|tail -n7
-r--r--r-- 1 gerrit2 gerrit2    2059 Jun 12 23:00 error_log.2024-06-11.gz
-r--r--r-- 1 gerrit2 gerrit2    2688 Jun 12 23:00 error_log.json.2024-06-11.gz
-r--r--r-- 1 gerrit2 gerrit2    5786 Jun 11 23:00 error_log.json.2024-06-10.gz
-r--r--r-- 1 gerrit2 gerrit2    4301 Jun 11 22:59 error_log.2024-06-10.gz
-rw-r--r-- 1 gerrit2 gerrit2       0 Feb  5 14:45 rename_log
-rw-r--r-- 1 gerrit2 gerrit2       0 May 25  2023 replication_log
-rw-r--r-- 1 gerrit2 gerrit2       0 May 25  2023 delete_log

Change #1049091 merged by Jelto:

[operations/puppet@production] gerrit: remove absented clear_gerrit_logs timer job

https://gerrit.wikimedia.org/r/1049091