Page MenuHomePhabricator

varnishkafka logrotate cronspam
Closed, ResolvedPublic

Description

This is cronspamming from every cp* host daily:

Date: Wed, 09 Mar 2016 06:25:06 +0000
From: Cron Daemon <root@cp4015.ulsfo.wmnet>
To: root@cp4015.ulsfo.wmnet
Subject: Cron <root@cp4015> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

/etc/cron.daily/logrotate:
Usage: /etc/init.d/rsyslog {start|stop|rotate|restart|force-reload|status}
error: error running non-shared postrotate script for /var/log/varnishkafka.log of '/var/log/varnishkafka.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

That's because /etc/logrotate.d/varnishkafka, shipped by the varnishkafka package (not puppet) calls service rsyslog reload which is an action that doesn't exist.

I'm unsure why this started happening all of a sudden — Brandon mentioned some logrotate permissions fix that was deployed recently that activated previously shipped logrotate configs, perhaps it's that.

The right invocation would probably be invoke-rc.d rsyslog rotate. None of the two belongs in the package though, as it creates an implicit dependency between the varnishkafka package and rsyslog which is configuration. The dependency apparently comes from an rsyslog config that we ship via puppet (files/varnish/varnishkafka_rsyslog.conf).

So:

  • Roll a new varnishkafka package without this logrotate config
  • Move that logrotate config to puppet
  • Replace service rsyslog reload with invoke-rc.d rsyslog rotate on that file
  • Check for other callsites of service rsyslog reload, e.g. I see modules/rsyslog/templates/receiver_logrotate.erb.conf

Related Objects

Event Timeline

faidon created this task.Mar 9 2016, 12:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 9 2016, 12:28 PM
elukey claimed this task.Mar 9 2016, 12:35 PM

Ariel submitted this https://gerrit.wikimedia.org/r/#/c/277220/ , but perhaps did not know about this ticket

elukey added a comment.EditedMar 14 2016, 5:54 PM

A bit of detail from cp1052:

elukey@cp1052:~$ ls -l /etc/logrotate.d/varnishkafka*
-r--r--r-- 1 root root 174 Mar  2 12:02 /etc/logrotate.d/varnishkafka
-r--r--r-- 1 root root 222 Aug 31  2015 /etc/logrotate.d/varnishkafka-eventlogging-stats
-r--r--r-- 1 root root 210 May 12  2015 /etc/logrotate.d/varnishkafka-statsv-stats
-r--r--r-- 1 root root 218 Mar 11  2015 /etc/logrotate.d/varnishkafka-webrequest-stats

Related config in puppet: class varnishkafka and varnishkafka::instance

file { '/etc/logrotate.d/varnishkafka':
    owner  => 'root',
    group  => 'root',
    mode   => '0444',
    source => 'puppet:///modules/varnishkafka/varnishkafka_logrotate'
}


file { "/etc/logrotate.d/varnishkafka-${name}-stats":
    owner   => 'root',
    group   => 'root',
    mode    => '0444',
    content => template('varnishkafka/varnishkafka-stats.logrotate.erb'),
    require => Package['varnishkafka'],
}

Also, last but not the least, the config file mentioned by Faidon.

So theoretically we could just remove rsyslog from the deb package and files/varnish and correct the rsyslog command in the puppet class (not sure if I am missing something though).

I am remembering things......

When @akosiaris and I packaged this thing, we had a lot of trouble getting logging with packaging to work properly. Perhaps you are right, that rsyslog stuff does not belong in the package, but we wanted a standalone varnishkafka log file, and there didn't seem to be an easy way to get varnishkafka (which was configured to log via generic syslog) to output to a separate log file without using rsyslog. This could be possibly wrong, but it is what I'm remembering.

files/varnish/varnishkafka_rsyslog.conf exists in puppet, because the rsyslog puppet module fully manages the /etc/rsyslog.d directory. If puppet didn't have a reference to this file, it would delete the file installed by the package. Previously we just declared the file and ensured it was present, but I think that caused some chicken v egg problems where sometimes the file would be deleted before puppet had a change to ensure it was present. Sticking the duplicate rsyslog.conf file into puppet and ensuring the content fixed this problem. Probably this file should be moved into the cache role module instead of top level files/.

I agree that this should be audited in the package, but I am guessing it might not be as easy as you say. Maybe we should just forgo logging to a specific file via syslog with the package?

In the meantime, the quick fix is to just fix the puppet managed logrotate files with invoke-rc.d rsyslog rotate.

Change 278750 had a related patch set uploaded (by Elukey):
Fix varnishkafka cronspam due to non existent rsyslog action.

https://gerrit.wikimedia.org/r/278750

Change 278750 merged by Elukey:
Fix varnishkafka cronspam due to non existent rsyslog action.

https://gerrit.wikimedia.org/r/278750

Change 278855 had a related patch set uploaded (by Elukey):
Update the varnishkafka module for https://gerrit.wikimedia.org/r/#/c/278750/1

https://gerrit.wikimedia.org/r/278855

Change 278855 merged by Elukey:
Update the varnishkafka module for https://gerrit.wikimedia.org/r/#/c/278750/1

https://gerrit.wikimedia.org/r/278855

Change 279308 had a related patch set uploaded (by Elukey):
Add basic varnishkafka rsyslog config to the varnishkafka module.

https://gerrit.wikimedia.org/r/279308

Proposal of next steps:

  1. move files/varnish/varnishkafka_rsyslog.conf to the varnishkafka module and force cache::kafka to reference it. This will remove a file from the puppet top level dirs. Moreover the varnishkafka module already contains basic logrotate config.
  1. Decide to remove or not the logrotate/rsyslog configs inside the package

Change 279308 merged by Elukey:
Add basic varnishkafka rsyslog config to the varnishkafka module.

https://gerrit.wikimedia.org/r/279308

Change 280678 had a related patch set uploaded (by Elukey):
Update the varnishkafka module with latest changes.

https://gerrit.wikimedia.org/r/280678

Change 280690 had a related patch set uploaded (by Elukey):
Add rsyslog configuration only if Service['rsyslog'] has been defined.

https://gerrit.wikimedia.org/r/280690

Change 280690 merged by Ottomata:
Add rsyslog configuration only if Service['rsyslog'] has been defined.

https://gerrit.wikimedia.org/r/280690

Change 280678 merged by Elukey:
Update the varnishkafka module with latest changes.

https://gerrit.wikimedia.org/r/280678

elukey added a comment.Apr 1 2016, 9:05 AM

Update:

  • rsyslog command has been updated to avoid cronspam
  • files/varnish/varnishkafka_rsyslog.conf has been moved from the puppet repo to the varnishkafka submodule (logrotate config was already there)
  • removed varnishkafka rsyslog references from cache::kafka role
  • added puppet banner to logrotate config file to make sure that it was managed by puppet and not by the vk package.

Still to do:

  • update the varnishkafka package to remove rsyslog/logrotate configs (if still needed)
ema moved this task from Triage to In Progress on the Traffic board.Apr 1 2016, 12:35 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.

@faidon do you prefer to upgrade the vk package anyway or the above solution would suffice for the moment?

The Varnishkafka package shipping an rsyslog config is a bug in itself, so we should queue a fix for it. As you've already handled the issue at hand via puppet, it's okay if we don't build/deploy the new package immediately, but leave it for whenever we do the next upgrade.

Change 282135 had a related patch set uploaded (by Elukey):
Remove logrotate/syslog configurations.

https://gerrit.wikimedia.org/r/282135

Change 282135 abandoned by Elukey:
Remove logrotate/syslog configurations.

Reason:
Needs a rebase but I can't figure out why, trying to send a new one.

https://gerrit.wikimedia.org/r/282135

Change 282161 had a related patch set uploaded (by Elukey):
Remove logrotate/syslog configurations.

https://gerrit.wikimedia.org/r/282161

Change 282188 had a related patch set uploaded (by Elukey):
Remove logrotate/syslog configurations.

https://gerrit.wikimedia.org/r/282188

Change 282188 abandoned by Elukey:
Remove logrotate/syslog configurations.

Reason:
missed to specify the remote branch, this is wrongly filed against master not debian :(

https://gerrit.wikimedia.org/r/282188

Change 282161 merged by Elukey:
Remove logrotate/syslog configurations.

https://gerrit.wikimedia.org/r/282161

Change 282876 had a related patch set uploaded (by Elukey):
Package latest upstream (1.9.0-1)

https://gerrit.wikimedia.org/r/282876

Change 282876 abandoned by Elukey:
Package latest upstream (1.9.0-1)

Reason:
/me writes 100 times "use --track debian when you package stuff"

https://gerrit.wikimedia.org/r/282876

Change 282878 had a related patch set uploaded (by Elukey):
Package latest upstream (1.9.0-1)

https://gerrit.wikimedia.org/r/282878

Change 282878 merged by Ema:
Package latest upstream (1.9.0-1)

https://gerrit.wikimedia.org/r/282878

root@carbon:~# reprepro ls varnishkafka
varnishkafka | 1.0.2-1 | precise-wikimedia | amd64, source
varnishkafka | 1.0.6-1 |  trusty-wikimedia | amd64, source
varnishkafka | 1.0.7-1 |  jessie-wikimedia | amd64, source
varnishkafka | 1.0.9-1 |  jessie-wikimedia | amd64, source

Packaged varnishkafka 1.0.9-1 with the latest changes.

Milimetric triaged this task as Medium priority.Apr 12 2016, 4:18 PM
elukey moved this task from In Progress to Done on the Analytics-Kanban board.Apr 13 2016, 10:25 AM
elukey added a subscriber: ema.

Installed on maps hosts by @ema, we will rollout the new version everywhere along wiht the Varnish 4 upgrade.

elukey closed this task as Resolved.Apr 14 2016, 7:24 AM