Page MenuHomePhabricator

Log file monitoring
Closed, DeclinedPublic

Description

The application in wmde/FundraisingFrontend does write quite some logging data to files. We do some stuff with it, too: rotating, moving to a central location, and some files are being encrypted. What we actually never do, is having a look. Although we can be quite confident with what the application does, we should be aware of errors happening in the application. In the worst case, there might be problems that keep people from donating.

Requirements for our logging solution are:

  • Use an existing solution, that is easy to set up and configure (e. g. LogStash)
  • Digestion/Throttling (to make sure, we don't receive a bunch of notifications for the same error)
  • Client-side error logging using a "catch-all" approach.
  • Alert/Log long running database queries

Notes
We first tried to use Graylog because it is a one-stop solution for log ingestion, search, visualization, and notification. However, we discovered that Graylog can't be configured through text files in an idempotent fashion, which does not fit with our "configuration as code" requirement. Also, the Graylog search interface is quite rudimentary. We'll migrate the existing Graylog setup to an ELK stack. We can intertwine the migration steps with necessary logging setup steps:

Event Timeline

Restricted Application added a project: WMDE-FUN-Team. · View Herald TranscriptJul 27 2018, 5:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gabriel-wmde updated the task description. (Show Details)Jan 9 2019, 3:57 PM
gabriel-wmde updated the task description. (Show Details)Sep 5 2019, 3:58 PM
gabriel-wmde closed this task as Declined.Feb 14 2020, 5:26 PM

This whole parent task is now made obsolete by T243203: Get familiar with errBit setup. Individual unresolved child tasks can still be picked as needed