Maps services should send their logs to logstash. The Kartotherian logstash dashboard show logs from maps-test servers, but not from new maps200? servers. This might be just because those new servers have not seen traffic, but we need to validate this, and fix it if needed.
|Resolved||None||T137848 Remove referrer check from varnish for maps cluster|
|Resolved||MSantos||T138028 EPIC: Enable maps on every WMF wiki|
|Resolved||Gehel||T133744 Epic: switch Maps to production status|
|Resolved||None||T137618 Send logs to logstash for maps services (katotherian, tilerator, tileratorui)|
The 3 services (kartotherian, tilerator, tileratorui) are all configured to send logs to logstash via Gelf (see /etc/(kartotherian|tilerator|tileratorui)/config.yaml. They are all configured to log only level=WARN and logs are at the moment empty.
@Yurik do you have an easy way to generate a WARN log for testing?
Starting kartotherian as my own user with modified configuration (enable debug level logging, change log file and port), logs are written correctly to file and to logstash. Permissions seems fine on /var/log/kartotherian.
Note: actual service is run via service-runner.js. I'm unsure of what this does, but that might be the difference.
FWIW, pretty much all node services are running in firejail, and as far as I am aware there have not been any issues with logging.
Could you test with a temporarily lowered logging level, using the production user?
I now see logs in logstash. I'm completely unsure why I did not before (I might have just been blind). I was probably confused by the fact that logs on disk are still invisible. It seems that firejail mounts a tmpfs on /var/log, which is hiding the logs from my shell.