@scfc that'd be covered by requirement #5 which would be a filesystem based ingress. lighttpd would just write to a file and that'll get shipped to ES somehow. There are many different log 'shippers' we can choose from for this.

In T127367#2041822, @scfc wrote:

Does ElasticSearch work for arbitrary programs? Is there a proof-of-concept for a lighttpd, etc. setup that logs to ElasticSearch?

In WMF production we ship logs from HHVM and Apache2 to Elasticsearch using rsyslog and Logstash as intermediaries (HHVM/Apache2 -> local rsyslog -> remote Logstash -> Elasticsearch). This sort of setup would probably work for lighttpd as well, but as @yuvipanda said there are numerous shipping solutions that can be examined.

Graylog (https://github.com/graylog2/graylog2-server, https://www.graylog.org/) seems to cover most of these points:

1. Take load off NFS - logs are stored on an Elasticsearch cluster
2. Make it far faster to see the actual logs from processes - doesn't depend on NFS, so should be fast
3. Be able to search through logs easier - searching is easy: http://docs.graylog.org/en/2.0/pages/queries.html, "The search syntax is very close to the Lucene syntax. By default all message fields are included in the search if you don’t specify a message field to search in."
4. Automatically drop older logs - index rotation can be configured based on message count, index size, or index time: http://docs.graylog.org/en/2.0/pages/index_model.html#eviction-of-indices-and-messages
5. Provide a Filesystem based interface for log ingress - Graylog supports this: http://docs.graylog.org/en/2.0/pages/sending_data.html#reading-from-files, "we provide the Collector Sidecar which acts as a supervisor process for other programs, such as nxlog and Filebeats, which have specifically been built to collect log messages from local files and ship them to remote systems like Graylog."
6. Provide more standard and modern interfaces (gelf? etc) for log ingress - Graylog supports GELF and syslog: http://docs.graylog.org/en/2.0/pages/sending_data.html
7. Provide a filesystem based interface for log reading - I don't think this is supported, but you can export search results to CSV: http://docs.graylog.org/en/2.0/pages/queries.html#export-results-as-csv
8. Provide a more modern interface for log reading as well - Graylog's interface looks fairly modern and easy to use to me:
   
   
9. Be secure in allowing only authenticated members to read a particular tool's logs. - Graylog has access control out-of-the-box, and can integrate with LDAP users and groups: http://docs.graylog.org/en/2.0/pages/users_and_roles/external_auth.html#ldap-active-directory

Graylog has Streams (basically categories for log messages): http://docs.graylog.org/en/2.0/pages/streams.html, alerting based on those streams: http://docs.graylog.org/en/2.0/pages/getting_started/stream_alerts.html, and dashboards: http://docs.graylog.org/en/2.0/pages/dashboards.html.

Thank you for finding this, and taking the time to work out all the points @tom29739! I think 7) is not too important -- the use case (as far as I can think of) is users who need to use output of a script as input to the next script. For those users, it's totally fine to just write to nfs.

In general, I think our first target is the webservice logs, and not so much the generic job output files -- those tend to not grow to as crazily large sizes.

[not a Tracking-Neverending bug per definition; maybe Epic or such was meant?]

T256426 is related to this.

Another related (maybe dependent?) task T293672

https://kubernetes.io/docs/concepts/cluster-administration/logging/ also contains a few pointers on how such architecture may be introduced on kubernetes.

linking this here: https://wikitech.wikimedia.org/wiki/User:Majavah/Loki_notes