terbium has a 500G disk with LVM so / can be extended:

$ lsblk 
NAME                                MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                   8:0    0 465.3G  0 disk 
├─sda1                                8:1    0   285M  0 part /boot
├─sda2                                8:2    0     1K  0 part 
└─sda5                                8:5    0   465G  0 part 
  ├─terbium--vg-root (dm-0)         252:0    0  67.3G  0 lvm  /
  ├─terbium--vg-swap_1 (dm-1)       252:1    0   7.5G  0 lvm  [SWAP]
  └─terbium--vg-_placeholder (dm-2) 252:2    0   4.7G  0 lvm  

nutcracker is --verbose=4 which matches the other application servers. The daemon default is 5 (from a range of 0 to 11).

Mentioned in SAL [2016-07-12T10:45:49Z] <godog> terbium:~# lvextend --size +70G -r /dev/mapper/terbium--vg-root T139786

I already solved the problem - nutcracker was running at verbosity 5. The only remaining problem is maybe getting rid of the old logfiles.

ok so now verbosity is 4, agreed there is no need to tune the log retention policy for nutcracker as the title suggests?

@Joe fixed the verbosity with: 07d8690 - nutcracker: lower verbosity on the maintenance hosts

Apparently we have overriden the value everywhere:

hieradata/role/common/mediawiki/appserver.yaml:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/appserver/api.yaml:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/appserver/canary_api.yaml:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/canary_appserver.yaml:nutcracker::verbosity: "4"        
hieradata/role/common/mediawiki/imagescaler.yaml:nutcracker::verbosity: "4"                                      
hieradata/role/common/mediawiki/jobrunner.yaml:nutcracker::verbosity: "4"                                        
hieradata/role/common/mediawiki/maintenance.yaml:nutcracker::verbosity: "4"                                      
hieradata/role/common/mediawiki/videoscaler.yaml:nutcracker::verbosity: "4"

The other uses I found are:

modules/role/manifests/labs/openstack/nova.pp : set to verbosity = 2

mediawiki::nutcracker which includes the nutcracker with the default verbosity. It is used by:

• role::deployment::mediawiki (eg: tin.eqiad.wmnet)
• snapshot and snapshot::dumps::packages

And deployment-prep which lacks the Hiera role hierarchy (T136078)

Since we tend to prefer verbosity=4 over 5, would it make sense to change our nutcracker puppet class to apply the sane default of 4 instead of 5?

class nutcracker(
    $pools,
    $ensure    = present,
    $mbuf_size = undef,
    $verbosity = 5,
) {

yeah I agree if the default verbosity of 5 (also nutcracker's) leads to useless/too verbose logging it should be lowered to 4 even in the module

Change 299146 had a related patch set uploaded (by Filippo Giunchedi):
nutcracker: default verbosity to 4

https://gerrit.wikimedia.org/r/299146

Change 299146 merged by Filippo Giunchedi:
nutcracker: default verbosity to 4

https://gerrit.wikimedia.org/r/299146