Page MenuHomePhabricator

On beta nutcracker::verbosity default to 5 while prod uses 4, fills disk with a lot of log spam
Closed, ResolvedPublic

Description

On the beta cluster, nutcracker logs at level 5 which is a lot of spam. Production has it set to 4:

hieradata/role/common/mediawiki/appserver.yaml:17:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/appserver/api.yaml:17:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/appserver/canary_api.yaml:22:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/canary_appserver.yaml:22:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/imagescaler.yaml:7:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/jobrunner.yaml:11:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/videoscaler.yaml:6:nutcracker::verbosity: "4"

The setting is not applied though because hieradata/role/* hierarchy is not supported on labs.

Event Timeline

hashar changed the task status from Open to Stalled.Jun 30 2016, 10:34 AM
hashar triaged this task as Low priority.

Blocked on T120165

Change 299146 had a related patch set uploaded (by Hashar):
nutcracker: default verbosity to 4

https://gerrit.wikimedia.org/r/299146

Change 299146 merged by Filippo Giunchedi:
nutcracker: default verbosity to 4

https://gerrit.wikimedia.org/r/299146

hashar claimed this task.

Fixed via https://gerrit.wikimedia.org/r/299146

-DAEMON_OPTS="--verbose=5 --mbuf-size=65536"
+DAEMON_OPTS="--verbose=4 --mbuf-size=65536"

Info: /Stage[main]/Nutcracker/File[/etc/default/nutcracker]: Scheduling refresh of Service[nutcracker]
Notice: /Stage[main]/Nutcracker/Service[nutcracker]: Triggered 'refresh' from 1 events

All beta cluster nutcracker demons now have verbose=4

# salt --out=text  -v '*' cmd.run 'grep verbose /etc/default/nutcracker'|grep verbose
mira.deployment-prep.eqiad.wmflabs: DAEMON_OPTS="--verbose=4 --mbuf-size=65536"
deployment-mediawiki01.deployment-prep.eqiad.wmflabs: DAEMON_OPTS="--verbose=4 --mbuf-size=65536"
deployment-mediawiki03.deployment-prep.eqiad.wmflabs: DAEMON_OPTS="--verbose=4 --mbuf-size=65536"
deployment-mediawiki02.deployment-prep.eqiad.wmflabs: DAEMON_OPTS="--verbose=4 --mbuf-size=65536"
deployment-tin.deployment-prep.eqiad.wmflabs: DAEMON_OPTS="--verbose=4 --mbuf-size=65536"
deployment-tmh01.deployment-prep.eqiad.wmflabs: DAEMON_OPTS="--verbose=4 --mbuf-size=65536"
deployment-jobrunner01.deployment-prep.eqiad.wmflabs: DAEMON_OPTS="--verbose=4 --mbuf-size=65536"