In T333143: Move Gerrit data out of root partition a lot of Gerrit-related data was moved out of the root partition into /srv/gerrit. During the migration the topic of logging, log-verbosity and logrotation came up. So we should double check how much logs are produced and stored in /var/logs and potential other places.
In general we should make sure logs are not filling up the root partition. In the past we had quite verbose logs (for example from nftables throttling).
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dzahn | T425667 Investigate Gerrit root disk usage and logging | |||
| Resolved | • hashar | T430774 Compact Gerrit H2 databases to reclaim disk |
Change #1298931 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: ensure error_log.json, sshd_log.json are always shipped to ELK
Change #1298932 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: adjust path to gc_log
Change #1298938 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: flip direction of symlink for log directories
Change #1298939 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: move httpd logs to $site_path/logs
Change #1298932 merged by Dzahn:
[operations/puppet@production] gerrit: adjust path to gc_log
Change #1298931 merged by Dzahn:
[operations/puppet@production] gerrit: ensure error_log.json, sshd_log.json are always shipped to ELK
Change #1298939 merged by Dzahn:
[operations/puppet@production] gerrit: move httpd logs to $site_path/logs
Change #1299642 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: fix variable name for site path used in httpd template
Change #1299642 merged by Dzahn:
[operations/puppet@production] gerrit: fix variable name for site path used in httpd template
Mentioned in SAL (#wikimedia-operations) [2026-06-09T22:07:08Z] <mutante> gerrit - apache httpd log file location moved to /srv/gerrit/site_path/review_site/logs/ T425667
Also made sure that Gerrit's own error_log.json and sshd_log.json are shipped to ELK (again).
And fixed the log path for the gc_log in gerrit-git-gc-timing.py.
It looks like that broke shipping of logs to logstash. Logs are missing since yesterday: https://logstash.wikimedia.org/goto/ca6430972b71af70684c34cfe55002c6. I'll upload a patch in a sec.
Change #1300049 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] gerrit: update rsyslog configuration for apache logs
Change #1300049 merged by Jelto:
[operations/puppet@production] gerrit: update rsyslog configuration for apache logs
Apache access and error logs are available again after merging the patch (also backfilled).
Change #1300187 had a related patch set uploaded (by Arnaudb; author: Arnaudb):
[operations/puppet@production] gerrit: point mtail at relocated httpd access logs
Change #1300187 merged by Arnaudb:
[operations/puppet@production] gerrit: point mtail at relocated httpd access logs
Change #1298938 merged by Dzahn:
[operations/puppet@production] gerrit: flip direction of symlink for log directories
Thanks for that fix @Jelto . That was on me for forgetting to also change that.
After the further change above which switched the symlink around and some manual action to move logs around we now have this:
root@gerrit2003:/srv/gerrit/site_path/review_site/logs# pwd -P /srv/gerrit/site_path/review_site/logs root@gerrit2003:/var/log/gerrit# pwd -P /srv/gerrit/site_path/review_site/logs
tail -f /srv/gerrit/site_path/review_site/logs/gerrit.wikimedia.org.https.access.log ...
gerrit2003:/srv/gerrit/site_path/review_site/logs# tail -f *.json
I claim everything that is left under /var/log now is not gerrit related and gerrit logs are not ending up in the root partition anymore.
root@gerrit2003:/var/log/gerrit# lsof /var/log/gerrit COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME bash 3519881 root cwd DIR 253,2 12288 40632345 /srv/gerrit/site_path/review_site/logs lsof 3714081 root cwd DIR 253,2 12288 40632345 /srv/gerrit/site_path/review_site/logs lsof 3714082 root cwd DIR 253,2 12288 40632345 /srv/gerrit/site_path/review_site/logs
/dev/mapper/vg0-root 93G 30G 59G 35% /
Noticed that the current apache log keeps on growing, seems like it's no longer logrotated (previously, under /var/log/apache they seem to rotate ... nightly? whatever, they're all gzip'd 300M).
Meanwhile:
root@gerrit2003:~# du -h /srv/gerrit/site_path/review_site/logs/gerrit.wikimedia.org.https.access.log 43G /srv/gerrit/site_path/review_site/logs/gerrit.wikimedia.org.https.access.log
Oldest entry there is from 4 days ago.
Change #1301462 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: add logrotate for httpd with custom log location
Change #1301462 merged by Dzahn:
[operations/puppet@production] gerrit: add logrotate for httpd with custom log location
Change #1302238 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gerrit: fix syntax for logrotate template content
Change #1302238 merged by Dzahn:
[operations/puppet@production] gerrit: fix syntax for logrotate template content
Thanks for reporting. I confirm there was no more logrotate under the new location. The httpd module always expects them in /var/log/apache2
On Saturday I manually added config to match the new path under /srv/ and otherwise use the same config used for httpd logs, then manually ran logrotate to ensure we would not run out of disk over the weekend for sure.
Now I merged the puppetized version of that. See above.
I notice now on gerrit2002 (the spare!) there is /srv/gerrit/site_path/review_site/logs/logs (with 2 logs at the end). Did anyone move those around?
anyways, no problem here right now for the prod server.
du -h /srv/gerrit/site_path/review_site/logs/ 14G /srv/gerrit/site_path/review_site/logs/ du: cannot access '/srv/gerrit/site_path/review_site/logs/logs': No such file or directory
5.1G gerrit.wikimedia.org.https.access.log
I am not sure yet where this needs to be fixed:
[sre-collaboration-services] [FIRING:1] AlertLintProblem collaboration-services (/srv/alerts/ops/team-collaboration-services_gerrit.yaml localhost:9123 pint alerting GerritHigh5xxRatio prometheus "ops" at http://127.0.0.1:9900/ops has "apache_http_status_total" metric but doesn't currently have series matching {status=~"5.*"}, such series was last present 6d22h ago ops promql/series warning eqiad prometheus)Mentioned in SAL (#wikimedia-operations) [2026-06-17T23:14:12Z] <mutante> gerrit2002 - unlink /srv/gerrit/site_path/review_site/logs/logs (T425667)
@ABran-WMF Is this alert above just telling us that there were NO 5xxs on one of the hosts? I checked with curl and while we have some 5xx on the prod machine there are none on replica and spare. Also, looking at the alerts dashboard there is no alert now.
I forgot to mention it, it's been fixed via this change
Indeed, # pint disable promql/series was missing in the first version of the alert so it was firing because there is little to no 5xx on the other hosts. Sorry for the noise!