Page MenuHomePhabricator

Rotate (nutcracker) logs more frequently on terbium to save disk space
Closed, ResolvedPublic

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 8 2016, 7:27 PM
hashar added a subscriber: hashar.Jul 8 2016, 9:55 PM

terbium has a 500G disk with LVM so / can be extended:

$ lsblk 
NAME                                MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                   8:0    0 465.3G  0 disk 
├─sda1                                8:1    0   285M  0 part /boot
├─sda2                                8:2    0     1K  0 part 
└─sda5                                8:5    0   465G  0 part 
  ├─terbium--vg-root (dm-0)         252:0    0  67.3G  0 lvm  /
  ├─terbium--vg-swap_1 (dm-1)       252:1    0   7.5G  0 lvm  [SWAP]
  └─terbium--vg-_placeholder (dm-2) 252:2    0   4.7G  0 lvm  

nutcracker is --verbose=4 which matches the other application servers. The daemon default is 5 (from a range of 0 to 11).

Mentioned in SAL [2016-07-12T10:45:49Z] <godog> terbium:~# lvextend --size +70G -r /dev/mapper/terbium--vg-root T139786

Joe added a subscriber: Joe.Jul 12 2016, 10:50 AM

I already solved the problem - nutcracker was running at verbosity 5. The only remaining problem is maybe getting rid of the old logfiles.

ok so now verbosity is 4, agreed there is no need to tune the log retention policy for nutcracker as the title suggests?

@Joe fixed the verbosity with: 07d8690 - nutcracker: lower verbosity on the maintenance hosts

Apparently we have overriden the value everywhere:

hieradata/role/common/mediawiki/appserver.yaml:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/appserver/api.yaml:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/appserver/canary_api.yaml:nutcracker::verbosity: "4"
hieradata/role/common/mediawiki/canary_appserver.yaml:nutcracker::verbosity: "4"        
hieradata/role/common/mediawiki/imagescaler.yaml:nutcracker::verbosity: "4"                                      
hieradata/role/common/mediawiki/jobrunner.yaml:nutcracker::verbosity: "4"                                        
hieradata/role/common/mediawiki/maintenance.yaml:nutcracker::verbosity: "4"                                      
hieradata/role/common/mediawiki/videoscaler.yaml:nutcracker::verbosity: "4"

The other uses I found are:

modules/role/manifests/labs/openstack/nova.pp : set to verbosity = 2

mediawiki::nutcracker which includes the nutcracker with the default verbosity. It is used by:

  • role::deployment::mediawiki (eg: tin.eqiad.wmnet)
  • snapshot and snapshot::dumps::packages

And deployment-prep which lacks the Hiera role hierarchy (T136078)

Since we tend to prefer verbosity=4 over 5, would it make sense to change our nutcracker puppet class to apply the sane default of 4 instead of 5?

class nutcracker(
    $pools,
    $ensure    = present,
    $mbuf_size = undef,
    $verbosity = 5,
) {

yeah I agree if the default verbosity of 5 (also nutcracker's) leads to useless/too verbose logging it should be lowered to 4 even in the module

Change 299146 had a related patch set uploaded (by Filippo Giunchedi):
nutcracker: default verbosity to 4

https://gerrit.wikimedia.org/r/299146

fgiunchedi triaged this task as Medium priority.Jul 15 2016, 2:47 PM

Change 299146 merged by Filippo Giunchedi:
nutcracker: default verbosity to 4

https://gerrit.wikimedia.org/r/299146

fgiunchedi closed this task as Resolved.Jul 20 2016, 9:36 AM
fgiunchedi claimed this task.
fgiunchedi added a subscriber: Mattflaschen-WMF.

I'm tentatively resolving this @Mattflaschen-WMF though the actual fix was defaulting to lower verbosity, not more aggressive log rotation policy