in Q1 2017-2018 a bullet point of a quarterly goal was **Standardize on a "default" pod setup** T170120. While the bullet (and the entire goal) was successfully completed, the end result for logging (which was log directly to logstash from the application) did have a number of drawbacks, most notably:
* Standard kubernetes tools like `kubectl logs` don't work with it
* It is more involved and complicated to differentiate between development and production environments as it needs extra configuration
* Moves a lot of the logic and responsibility of logging into each application, making divergence more probable and adding complexity (and bugs) into the application.
* While a supported configuration, it is not one used by many in the kubernetes community, meaning it's a tad more difficult to have support for it
Back then `fluentd` was evaluated, and containers built and tried. Unfortunately, due to changes and restructuring in the fluentd software some functionality was not working
In the meantime, Infrastructure Foundations has adopted the logging infrastructure and is moving forward with aa redesign of it in T205849. The recommended solution for log shipping in this is using rsyslog which does support kubernetes.
This task is about:
[x] Evaluating rsyslog (with the mmkubernetes plugin) vs (newer version of) fluentd
[] Settling down on one of the two for shipping logs from applications in our kubernetes environments
[] Implementing the solution