Page MenuHomePhabricator

api-gateway should ship logs to logstash
Open, Needs TriagePublic

Description

Currently the only way to debug issues with the API gateway is to consume logs directly from Kubernetes. This isn't ideal as it's quite manual and not good for long term issues. The API Gateway needs to ship application logs as JSON to logstash - currently this is not supported by Envoy. While doing this we should also try to tune Envoy logs as they are currently extremely verbose for every request and even for background information.

Event Timeline

Ran into this issue today while digging into log spikes from api-gateway (T295717). Envoy offers --log-format and --log-format-escaped (link) which could help us either populate a JSON message or at the very least escape the newline characters.