The application in wmde/FundraisingFrontend does write quite some logging data to files. We do some stuff with it, too: rotating, moving to a central location, and some files are being encrypted. What we actually never do, is having a look. Although we can be quite confident with what the application does, we should be aware of errors happening in the application. In the worst case, there might be problems that keep people from donating.
Requirements for our logging solution are:
- Use an existing solution, that is easy to set up and configure (e. g. LogStash)
- Digestion/Throttling (to make sure, we don't receive a bunch of notifications for the same error)
- Client-side error logging using a "catch-all" approach.
- Alert/Log long running database queries
We first tried to use Graylog because it is a one-stop solution for log ingestion, search, visualization, and notification. However, we discovered that Graylog can't be configured through text files in an idempotent fashion, which does not fit with our "configuration as code" requirement. Also, the Graylog search interface is quite rudimentary. We'll migrate the existing Graylog setup to an ELK stack. We can intertwine the migration steps with necessary logging setup steps:
- T203692: Log PHP errors This is a "proof of concept" that introduces Logstash and Filebeat.
- T213294: Set up Kibana This sets up an alternative UI to Graylog, reducing Graylog to the role of a log processor for the application log events
- T213293: Use files for production logging, remove Graylog This is the final step for getting rid of Graylog
- T213297: Update ELK Stack