Page MenuHomePhabricator
Create Task
Maniphest T144797

graphite-web cronspam
Closed, ResolvedPublic
Actions

Description

Daily cronspam like:

/etc/cron.daily/logrotate:
uwsgi-graphite-web stop/waiting
uwsgi-graphite-web start/running, process 15986
gzip: stdin: file size changed while zipping
gzip: stdin: file size changed while zipping
gzip: stdin: file size changed while zipping
gzip: stdin: file size changed while zipping

Last attempt to fix the cronspam: https://gerrit.wikimedia.org/r/308526

The main issue seems to be that graphite-web does its own internal log rotation, that as far as I can see is not configurable until version 0.10 (meanwhile we run 0.9.13+debian-1):

https://github.com/graphite-project/graphite-web/blob/master/docs/releases/0_10_0.rst

Pythons own log rotation can be disabled using the LOG_ROTATION setting. This is useful in case of running multiple WSGI workers. See also: Github Issue #499.

Maybe disabling compression could help?

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -1Reduce cronspam from graphite1001
operations/puppetproduction+9 -6Test graphite-web's internal logrotation
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT132324 Tracking and Reducing cron-spam to root@
ResolvedelukeyT144797 graphite-web cronspam

Event Timeline

elukey created this task.Sep 6 2016, 9:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 6 2016, 9:44 AM
elukey added a subscriber: fgiunchedi.Sep 8 2016, 7:06 AM

Checked also https://github.com/graphite-project/graphite-web/blob/0.9.13-pre1/webapp/graphite/logger.py

These are the rotation rules:

#Setup formatter & handlers
self.formatter = logging.Formatter("%(asctime)s :: %(message)s","%a %b %d %H:%M:%S %Y")
self.infoHandler = Rotater(self.infoLogFile,when="midnight",backupCount=1)
self.infoHandler.setFormatter(self.formatter)
self.infoLogger.addHandler(self.infoHandler)
self.exceptionHandler = Rotater(self.exceptionLogFile,when="midnight",backupCount=1)
self.exceptionHandler.setFormatter(self.formatter)
self.exceptionLogger.addHandler(self.exceptionHandler)
if settings.LOG_CACHE_PERFORMANCE:
  self.cacheHandler = Rotater(self.cacheLogFile,when="midnight",backupCount=1)
  self.cacheHandler.setFormatter(self.formatter)
  self.cacheLogger.addHandler(self.cacheHandler)
if settings.LOG_RENDERING_PERFORMANCE:
  self.renderingHandler = Rotater(self.renderingLogFile,when="midnight",backupCount=1)
  self.renderingHandler.setFormatter(self.formatter)
  self.renderingLogger.addHandler(self.renderingHandler)
if settings.LOG_METRIC_ACCESS:
  self.metricAccessHandler = Rotater(self.metricAccessLogFile,when="midnight",backupCount=10)
  self.metricAccessHandler.setFormatter(self.formatter)
  self.metricAccessLogger.addHandler(self.metricAccessHandler)

There is no cap on file size but it could be enough for our use case. Maybe we could evaluate graphite-web's own log rotation for a couple of days removing the logrotate script?

@fgiunchedi any thoughts?

elukey added a comment.Sep 9 2016, 9:56 AM

We could also think to add https://github.com/graphite-project/graphite-web/commit/78cee79623ffc4c89665c072aa8948d6bd2927f8 to the package but afaics we should also create a jessie-wikimedia version.

gerritbot added a subscriber: gerritbot.Sep 9 2016, 10:04 AM

Change 309544 had a related patch set uploaded (by Elukey):
Test graphite-web's internal logrotation

https://gerrit.wikimedia.org/r/309544

gerritbot added a project: Patch-For-Review.Sep 9 2016, 10:05 AM
gerritbot added a comment.Sep 9 2016, 10:30 AM

Change 309544 abandoned by Elukey:
Test graphite-web's internal logrotation

Reason:
After a chat with Filippo we discovered that the code on the package is not the one that I expected, so this patch needs to hold

https://gerrit.wikimedia.org/r/309544

elukey added a comment.Sep 9 2016, 10:38 AM

I was wrong:

http://launchpadlibrarian.net/207471033/graphite-web_0.9.12+debian-7_0.9.13+debian-1.diff.gz

in the package there is a patch called "remove_internal_logrotate.patch", so this is not the root cause of the cronspam.

gerritbot added a comment.Sep 30 2016, 10:14 AM

Change 313573 had a related patch set uploaded (by Elukey):
Reduce cronspam from graphite1001

https://gerrit.wikimedia.org/r/313573

elukey added a comment.Sep 30 2016, 12:43 PM

Two code reviews to review/merge:

gerritbot added a comment.Oct 3 2016, 7:33 AM

Change 313573 merged by Elukey:
Reduce cronspam from graphite1001

https://gerrit.wikimedia.org/r/313573

elukey closed this task as Resolved.Oct 5 2016, 12:40 PM
elukey claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL