Currently, tools just default to writing log files on to NFS. While simple, this causes a number of problems:
1. It adds additional load on our NFS server, which isn't already doing great
2. There's a delay between the logs being written on the exec node and being readable on bastion, which is both confusing and annoying
3. Logrotate is a PITA with GridEngine + NFS
A solution (based on ElasticSearch, probably - to mirror what we have in production), should allow us to do the following:
1. Take load off NFS
2. Make it far faster to see the actual logs from processes
3. Be able to search through logs easier
4. Automatically drop older logs
5. Provide a Filesystem based interface for log ingress
6. Provide more standard and modern interfaces (gelf? etc) for log ingress
7. Provide a filesystem based interface for log reading
8. Provide a more modern interface for log reading as well
9. Be secure in allowing only authenticated members to read a particular tool's logs.
This is the tracking ticket for this overhaul.
This is specifically *only* for Tool Labs, and not for use by general labs, mostly due to concern 9.
Other related tasks:
- {T50846} (now defunct)
- {T97861}
- {T127368}