Page MenuHomePhabricator

Send logs to logstash for maps services (katotherian, tilerator, tileratorui)
Closed, ResolvedPublic

Description

Maps services should send their logs to logstash. The Kartotherian logstash dashboard show logs from maps-test servers, but not from new maps200? servers. This might be just because those new servers have not seen traffic, but we need to validate this, and fix it if needed.

Details

Related Gerrit Patches:
operations/puppet : productionRevert "Move service logs to /srv/log"
operations/puppet : productionMove service logs to /srv/log

Event Timeline

Gehel created this task.Jun 11 2016, 6:46 AM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 11 2016, 6:46 AM
Gehel added a comment.Jun 11 2016, 7:05 PM

The 3 services (kartotherian, tilerator, tileratorui) are all configured to send logs to logstash via Gelf (see /etc/(kartotherian|tilerator|tileratorui)/config.yaml. They are all configured to log only level=WARN and logs are at the moment empty.

@Yurik do you have an easy way to generate a WARN log for testing?

Gehel moved this task from Backlog to In progress on the Maps-Sprint board.
Gehel added a comment.Jun 13 2016, 3:56 PM

Starting kartotherian as my own user with modified configuration (enable debug level logging, change log file and port), logs are written correctly to file and to logstash. Permissions seems fine on /var/log/kartotherian.

Note: actual service is run via service-runner.js. I'm unsure of what this does, but that might be the difference.

Gehel added a comment.Jun 13 2016, 6:43 PM

service-runner.js is run in the same way in my test and for actual service. Difference might be firejail... needs investigation.

FWIW, pretty much all node services are running in firejail, and as far as I am aware there have not been any issues with logging.

Could you test with a temporarily lowered logging level, using the production user?

I now see logs in logstash. I'm completely unsure why I did not before (I might have just been blind). I was probably confused by the fact that logs on disk are still invisible. It seems that firejail mounts a tmpfs on /var/log, which is hiding the logs from my shell.

Change 294285 had a related patch set uploaded (by Gehel):
Move service logs to /srv/log

https://gerrit.wikimedia.org/r/294285

Change 294285 merged by Gehel:
Move service logs to /srv/log

https://gerrit.wikimedia.org/r/294285

Change 294286 had a related patch set uploaded (by Gehel):
Revert "Move service logs to /srv/log"

https://gerrit.wikimedia.org/r/294286

Change 294286 merged by Gehel:
Revert "Move service logs to /srv/log"

https://gerrit.wikimedia.org/r/294286

Gehel closed this task as Resolved.Jun 14 2016, 1:44 PM