Python has a very configurable logging system. Find out from Observability folks how they would like a Python app in the Kubernetes cluster to output logs such that they eventually end up in our production log aggregation system.
|Open||None||T288685 Establish active/active multi-dc support for Toolhub|
|Resolved||bd808||T115650 Create an authoritative and well promoted catalog of Wikimedia tools|
|Open||bd808||T271483 Complete and announce initial production deployment of Toolhub|
|Resolved||bd808||T276374 Figure out what production logging config needs to look like|
@bd808. I think that if your logging is compatible to ecs logging schema (see https://github.com/elastic/ecs-logging) it will be quite ok. The design doc and reasoning from the observability team can be found at https://docs.google.com/document/d/1HYHCPvuz93nAYXQSEReUN07HQTQUF_nvltag5H_YZq4/edit#
Trying out https://github.com/elastic/ecs-logging-python has been "fun". The library assumes that any and all values placed into the "extras" of a logging event is JSON serializable content. My first page load using that formatter proved otherwise when the wsgi access log event blew up due to a socket.socket object being present in the logging context data. I'm not at all sure that I will be able to bend all of the stock Python + Django logging to fit the contrived schema that is https://github.com/elastic/ecs-logging. The upstream library does not seem to provide any tools for doing this. It seems mostly oriented towards green field development where you have no 3rd party/non-ECS aware log message generation.
Yes, and that's actually the most difficult part in an application like Toolforge which is using multiple upstream libraries and frameworks. Using the core fields are simple enough, but coercing all structured data to fit the extensions is the part that I'm skeptical of.
I was wondering what needed to be done for log shipping, but @Legoktm pointed me to T207200: Revisit the logging work done on Q1 2017-2018 for the standard pod setup which seems to indicate that the stderr logging will be picked up on the exec nodes and shipped to the ELK stack for me by some fancy rsyslog magic.