The 3 services (kartotherian, tilerator, tileratorui) are all configured to send logs to logstash via Gelf (see /etc/(kartotherian|tilerator|tileratorui)/config.yaml. They are all configured to log only level=WARN and logs are at the moment empty.

@Yurik do you have an easy way to generate a WARN log for testing?

Starting kartotherian as my own user with modified configuration (enable debug level logging, change log file and port), logs are written correctly to file and to logstash. Permissions seems fine on /var/log/kartotherian.

Note: actual service is run via service-runner.js. I'm unsure of what this does, but that might be the difference.

service-runner.js is run in the same way in my test and for actual service. Difference might be firejail... needs investigation.

FWIW, pretty much all node services are running in firejail, and as far as I am aware there have not been any issues with logging.

Could you test with a temporarily lowered logging level, using the production user?

I now see logs in logstash. I'm completely unsure why I did not before (I might have just been blind). I was probably confused by the fact that logs on disk are still invisible. It seems that firejail mounts a tmpfs on /var/log, which is hiding the logs from my shell.

Change 294285 had a related patch set uploaded (by Gehel):
Move service logs to /srv/log

https://gerrit.wikimedia.org/r/294285

Change 294285 merged by Gehel:
Move service logs to /srv/log

https://gerrit.wikimedia.org/r/294285

Change 294286 had a related patch set uploaded (by Gehel):
Revert "Move service logs to /srv/log"

https://gerrit.wikimedia.org/r/294286

Change 294286 merged by Gehel:
Revert "Move service logs to /srv/log"

https://gerrit.wikimedia.org/r/294286