I spoke to @Legoktm and it'd appear there's no logrotate or alternative available for the Toolforge Jobs framework/k8s (apologies if I'm mistaken).
In order to better manage logs, could it be possible to have logrotate installed in the toolforge jobs framework?
Thank you.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Bstorm | T126083 overhaul labstore setup [tracking] | |||
| Resolved | • GTirloni | T216988 labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%): | |||
| Resolved | • Bstorm | T217993 2019-03-10: tools and NFS share cleanup (high usage) | |||
| Resolved | • Bstorm | T122508 Prevent overly-large log files | |||
| Declined | None | T286847 Add webservice flag to mount project directory read-only | |||
| Open | None | T127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge webservices and bots | |||
| Resolved | None | T327165 toolforge-jobs: add logrotate |
logrotate is available in the mariadb image created for T254636: mysqldump is not present in Kubernetes container images (along with mariadb cli clients and our local sql wrapper).
How/where should we document that? It would be neat to figure out how to manage an automated inventory of what packages are available in which images.
If you've discovered a hack that allows logrotate to be used on the Jobs framework, can you please document that HERE, where it says:
NOTE: as of this writing there is no automatic way to prune log files, so tool users must take care of such files growing too large.
and:
Please, note that providing more modern approaches and facilities for logs management, metrics, etc. is in the current roadmap for the WMCS team. See Phabricator T127367 for example.
T127367 says (problem #3:)
Logrotate is a PITA with GridEngine + NFS
But, I suppose having a solution which is a PITA is better than having no solution at all.
I've never used logrotate before and am not familiar with how it works. Please do tell us how you brute-forced it. Thanks!
My 'brute-force' method was pretty simple. I manually tried different images one by one and eventually found one that doesn't say logrotate is not found when executing "logrotate --version". Maybe I just got lucky when I hit it fairly soon. "Find an image that doesn't fail at command X" could perhaps be made as a script, though.
Oh, I see (lightbulb turning on):
tools.billsbots@tools-sgebastion-10:~$ toolforge-jobs run logrotate --command "logrotate --version" --image mariadb
produces logrotate.out:
logrotate 3.21.0
Default mail command: /usr/bin/mail
Default compress command: /bin/gzip
Default uncompress command: /bin/gunzip
Default compress extension: .gz
Default state file path: /var/lib/logrotate/status
ACL support: yes
SELinux support: yesI think maybe I can work with that to hack up a continuous toolforge-jobs job that performs ongoing log-file rotation...
I found this helpful documentation:
A Complete Guide to Managing Log Files with Logrotate
I tried setting my rotation up to use copytruncate mode. It's rotating, but I'm not sure whether it's actually truncating, and if it is, by how much.
OK, I've successfully set up log rotation for a couple of my Toolforge jobs, and have documented how I did it.