Graylog (https://github.com/graylog2/graylog2-server, https://www.graylog.org/) seems to cover most of these points:

1. Take load off NFS - logs are stored on an Elasticsearch cluster
2. Make it far faster to see the actual logs from processes - doesn't depend on NFS, so should be fast
3. Be able to search through logs easier - searching is easy: http://docs.graylog.org/en/2.0/pages/queries.html, "The search syntax is very close to the Lucene syntax. By default all message fields are included in the search if you don’t specify a message field to search in."
4. Automatically drop older logs - index rotation can be configured based on message count, index size, or index time: http://docs.graylog.org/en/2.0/pages/index_model.html#eviction-of-indices-and-messages
5. Provide a Filesystem based interface for log ingress - Graylog supports this: http://docs.graylog.org/en/2.0/pages/sending_data.html#reading-from-files, "we provide the Collector Sidecar which acts as a supervisor process for other programs, such as nxlog and Filebeats, which have specifically been built to collect log messages from local files and ship them to remote systems like Graylog."
6. Provide more standard and modern interfaces (gelf? etc) for log ingress - Graylog supports GELF and syslog: http://docs.graylog.org/en/2.0/pages/sending_data.html
7. Provide a filesystem based interface for log reading - I don't think this is supported, but you can export search results to CSV: http://docs.graylog.org/en/2.0/pages/queries.html#export-results-as-csv
8. Provide a more modern interface for log reading as well - Graylog's interface looks fairly modern and easy to use to me:
   
   
9. Be secure in allowing only authenticated members to read a particular tool's logs. - Graylog has access control out-of-the-box, and can integrate with LDAP users and groups: http://docs.graylog.org/en/2.0/pages/users_and_roles/external_auth.html#ldap-active-directory

Graylog has Streams (basically categories for log messages): http://docs.graylog.org/en/2.0/pages/streams.html, alerting based on those streams: http://docs.graylog.org/en/2.0/pages/getting_started/stream_alerts.html, and dashboards: http://docs.graylog.org/en/2.0/pages/dashboards.html.

Thank you for finding this, and taking the time to work out all the points @tom29739! I think 7) is not too important -- the use case (as far as I can think of) is users who need to use output of a script as input to the next script. For those users, it's totally fine to just write to nfs.

In general, I think our first target is the webservice logs, and not so much the generic job output files -- those tend to not grow to as crazily large sizes.

[not a Tracking-Neverending bug per definition; maybe Epic or such was meant?]

T256426 is related to this.

Another related (maybe dependent?) task T293672

https://kubernetes.io/docs/concepts/cluster-administration/logging/ also contains a few pointers on how such architecture may be introduced on kubernetes.

linking this here: https://wikitech.wikimedia.org/wiki/User:Majavah/Loki_notes

For anyone curious

for namespace in $(kubectl get ns | tail -n +2 | awk '{print $1}') ;
do
    for pod in $(kubectl get pods -n ${namespace} | tail -n +2 | awk '{print $1}') ;
    do
        kubectl -n ${namespace} logs ${pod} --all-containers --since=24h
    done
done

Suggests that there are about 500 megabytes of logs in the last twenty four hours. Suggesting that a monolithic loki could work.

In T127367#10406539, @rook wrote:

For anyone curious

for namespace in $(kubectl get ns | tail -n +2 | awk '{print $1}') ;
do
    for pod in $(kubectl get pods -n ${namespace} | tail -n +2 | awk '{print $1}') ;
    do
        kubectl -n ${namespace} logs ${pod} --all-containers --since=24h
    done
done

Suggests that there are about 500 megabytes of logs in the last twenty four hours. Suggesting that a monolithic loki could work.

I fear that this query is missing most of the logs. It ignores most toolforge-jobs jobs that are configured to log onto an NFS file, as well as any cron jobs that are not running at that specific time.

Fair enough, do you have any estimate on how much those logs would account for in a day?