Page MenuHomePhabricator

Provide modern, non-NFS error log solution for Toolforge webservices and bots
Open, HighPublic

Description

Currently, tools just default to writing log files on to NFS. While simple, this causes a number of problems:

  1. It adds additional load on our NFS server, which isn't already doing great
  2. There's a delay between the logs being written on the exec node and being readable on bastion, which is both confusing and annoying
  3. Logrotate is a PITA with GridEngine + NFS

A solution (based on ElasticSearch, probably - to mirror what we have in production), should allow us to do the following:

  1. Take load off NFS
  2. Make it far faster to see the actual logs from processes
  3. Be able to search through logs easier
  4. Automatically drop older logs
  5. Provide a Filesystem based interface for log ingress
  6. Provide more standard and modern interfaces (gelf? etc) for log ingress
  7. Provide a filesystem based interface for log reading
  8. Provide a more modern interface for log reading as well
  9. Be secure in allowing only authenticated members to read a particular tool's logs.

This is the tracking ticket for this overhaul.

This is specifically *only* for Toolforge, and not for use by general Cloud VPS projects, mostly due to concern 9.

Other related tasks:

Event Timeline

Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 18 2016, 9:19 PM
scfc added a subscriber: scfc.Feb 18 2016, 9:47 PM

Does ElasticSearch work for arbitrary programs? Is there a proof-of-concept for a lighttpd, etc. setup that logs to ElasticSearch?

@scfc that'd be covered by requirement #5 which would be a filesystem based ingress. lighttpd would just write to a file and that'll get shipped to ES somehow. There are many different log 'shippers' we can choose from for this.

Does ElasticSearch work for arbitrary programs? Is there a proof-of-concept for a lighttpd, etc. setup that logs to ElasticSearch?

In WMF production we ship logs from HHVM and Apache2 to Elasticsearch using rsyslog and Logstash as intermediaries (HHVM/Apache2 -> local rsyslog -> remote Logstash -> Elasticsearch). This sort of setup would probably work for lighttpd as well, but as @yuvipanda said there are numerous shipping solutions that can be examined.

chasemp triaged this task as High priority.Mar 18 2016, 2:38 PM
tom29739 added a subscriber: tom29739.
bd808 moved this task from Triage to Tracking on the Toolforge board.May 25 2016, 4:23 PM
valhallasw updated the task description. (Show Details)May 27 2016, 4:36 PM
valhallasw updated the task description. (Show Details)
bd808 updated the task description. (Show Details)May 27 2016, 4:38 PM

Graylog (https://github.com/graylog2/graylog2-server, https://www.graylog.org/) seems to cover most of these points:

  1. Take load off NFS - logs are stored on an Elasticsearch cluster
  2. Make it far faster to see the actual logs from processes - doesn't depend on NFS, so should be fast
  3. Be able to search through logs easier - searching is easy: http://docs.graylog.org/en/2.0/pages/queries.html, "The search syntax is very close to the Lucene syntax. By default all message fields are included in the search if you don’t specify a message field to search in."
  4. Automatically drop older logs - index rotation can be configured based on message count, index size, or index time: http://docs.graylog.org/en/2.0/pages/index_model.html#eviction-of-indices-and-messages
  5. Provide a Filesystem based interface for log ingress - Graylog supports this: http://docs.graylog.org/en/2.0/pages/sending_data.html#reading-from-files, "we provide the Collector Sidecar which acts as a supervisor process for other programs, such as nxlog and Filebeats, which have specifically been built to collect log messages from local files and ship them to remote systems like Graylog."
  6. Provide more standard and modern interfaces (gelf? etc) for log ingress - Graylog supports GELF and syslog: http://docs.graylog.org/en/2.0/pages/sending_data.html
  7. Provide a filesystem based interface for log reading - I don't think this is supported, but you can export search results to CSV: http://docs.graylog.org/en/2.0/pages/queries.html#export-results-as-csv
  8. Provide a more modern interface for log reading as well - Graylog's interface looks fairly modern and easy to use to me:
  9. Be secure in allowing only authenticated members to read a particular tool's logs. - Graylog has access control out-of-the-box, and can integrate with LDAP users and groups: http://docs.graylog.org/en/2.0/pages/users_and_roles/external_auth.html#ldap-active-directory

Graylog has Streams (basically categories for log messages): http://docs.graylog.org/en/2.0/pages/streams.html, alerting based on those streams: http://docs.graylog.org/en/2.0/pages/getting_started/stream_alerts.html, and dashboards: http://docs.graylog.org/en/2.0/pages/dashboards.html.

Thank you for finding this, and taking the time to work out all the points @tom29739! I think 7) is not too important -- the use case (as far as I can think of) is users who need to use output of a script as input to the next script. For those users, it's totally fine to just write to nfs.

In general, I think our first target is the webservice logs, and not so much the generic job output files -- those tend to not grow to as crazily large sizes.

[not a Tracking-Neverending bug per definition; maybe Epic or such was meant?]

bd808 renamed this task from Overhaul logging setup for Tools (Tracking) to Provide modern, non-NFS error log solution for Toolforge webservices and bots.Nov 27 2018, 11:38 PM
bd808 added a project: Epic.
bd808 updated the task description. (Show Details)
GTirloni removed a subscriber: GTirloni.Mar 21 2019, 9:06 PM