Page MenuHomePhabricator

graphite-web cronspam
Closed, ResolvedPublic

Description

Daily cronspam like:

/etc/cron.daily/logrotate:
uwsgi-graphite-web stop/waiting
uwsgi-graphite-web start/running, process 15986
gzip: stdin: file size changed while zipping
gzip: stdin: file size changed while zipping
gzip: stdin: file size changed while zipping
gzip: stdin: file size changed while zipping

Last attempt to fix the cronspam: https://gerrit.wikimedia.org/r/308526

The main issue seems to be that graphite-web does its own internal log rotation, that as far as I can see is not configurable until version 0.10 (meanwhile we run 0.9.13+debian-1):

https://github.com/graphite-project/graphite-web/blob/master/docs/releases/0_10_0.rst

Pythons own log rotation can be disabled using the LOG_ROTATION setting. This is useful in case of running multiple WSGI workers. See also: Github Issue #499.

Maybe disabling compression could help?

Related Objects

StatusSubtypeAssignedTask
OpenNone
Resolvedelukey

Event Timeline

elukey created this task.Sep 6 2016, 9:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 6 2016, 9:44 AM

Checked also https://github.com/graphite-project/graphite-web/blob/0.9.13-pre1/webapp/graphite/logger.py

These are the rotation rules:

#Setup formatter & handlers
self.formatter = logging.Formatter("%(asctime)s :: %(message)s","%a %b %d %H:%M:%S %Y")
self.infoHandler = Rotater(self.infoLogFile,when="midnight",backupCount=1)
self.infoHandler.setFormatter(self.formatter)
self.infoLogger.addHandler(self.infoHandler)
self.exceptionHandler = Rotater(self.exceptionLogFile,when="midnight",backupCount=1)
self.exceptionHandler.setFormatter(self.formatter)
self.exceptionLogger.addHandler(self.exceptionHandler)
if settings.LOG_CACHE_PERFORMANCE:
  self.cacheHandler = Rotater(self.cacheLogFile,when="midnight",backupCount=1)
  self.cacheHandler.setFormatter(self.formatter)
  self.cacheLogger.addHandler(self.cacheHandler)
if settings.LOG_RENDERING_PERFORMANCE:
  self.renderingHandler = Rotater(self.renderingLogFile,when="midnight",backupCount=1)
  self.renderingHandler.setFormatter(self.formatter)
  self.renderingLogger.addHandler(self.renderingHandler)
if settings.LOG_METRIC_ACCESS:
  self.metricAccessHandler = Rotater(self.metricAccessLogFile,when="midnight",backupCount=10)
  self.metricAccessHandler.setFormatter(self.formatter)
  self.metricAccessLogger.addHandler(self.metricAccessHandler)

There is no cap on file size but it could be enough for our use case. Maybe we could evaluate graphite-web's own log rotation for a couple of days removing the logrotate script?

@fgiunchedi any thoughts?

elukey added a comment.Sep 9 2016, 9:56 AM

We could also think to add https://github.com/graphite-project/graphite-web/commit/78cee79623ffc4c89665c072aa8948d6bd2927f8 to the package but afaics we should also create a jessie-wikimedia version.

Change 309544 had a related patch set uploaded (by Elukey):
Test graphite-web's internal logrotation

https://gerrit.wikimedia.org/r/309544

Change 309544 abandoned by Elukey:
Test graphite-web's internal logrotation

Reason:
After a chat with Filippo we discovered that the code on the package is not the one that I expected, so this patch needs to hold

https://gerrit.wikimedia.org/r/309544

I was wrong:

http://launchpadlibrarian.net/207471033/graphite-web_0.9.12+debian-7_0.9.13+debian-1.diff.gz

in the package there is a patch called "remove_internal_logrotate.patch", so this is not the root cause of the cronspam.

Change 313573 had a related patch set uploaded (by Elukey):
Reduce cronspam from graphite1001

https://gerrit.wikimedia.org/r/313573

Change 313573 merged by Elukey:
Reduce cronspam from graphite1001

https://gerrit.wikimedia.org/r/313573

elukey closed this task as Resolved.Oct 5 2016, 12:40 PM
elukey claimed this task.