Page MenuHomePhabricator

cronspam from cpXXXX hosts related to varnishkafka non existent processes
Closed, ResolvedPublic

Description

Cron Daemon root@cp4019.ulsfo.wmnet via wikimedia.org 

to root 
/etc/cron.daily/logrotate:
Job for varnishkafka-eventlogging.service failed. See 'systemctl status varnishkafka-eventlogging.service' and 'journalctl -xn' for details.
error: error running non-shared postrotate script for /var/cache/varnishkafka/eventlogging.stats.json of '/var/cache/varnishkafka/eventlogging.stats.json '
Job for varnishkafka-statsv.service failed. See 'systemctl status varnishkafka-statsv.service' and 'journalctl -xn' for details.
error: error running non-shared postrotate script for /var/cache/varnishkafka/statsv.stats.json of '/var/cache/varnishkafka/statsv.stats.json '
Job for varnishkafka-webrequest.service failed. See 'systemctl status varnishkafka-webrequest.service' and 'journalctl -xn' for details.
error: error running non-shared postrotate script for /var/cache/varnishkafka/webrequest.stats.json of '/var/cache/varnishkafka/webrequest.stats.json '
elukey@cp4019:~$ systemctl status varnishkafka-eventlogging.service
● varnishkafka-eventlogging.service - VarnishKafka eventlogging
   Loaded: loaded (/lib/systemd/system/varnishkafka-eventlogging.service; enabled)
   Active: inactive (dead) since Mon 2016-02-08 15:23:14 UTC; 2 months 2 days ago
 Main PID: 1928 (code=exited, status=0/SUCCESS)

This seems to be related to a cron installed on cp hosts not running varnishkafka.

Event Timeline

elukey created this task.Apr 11 2016, 3:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 11 2016, 3:47 PM
ema triaged this task as Medium priority.Apr 12 2016, 6:01 PM
elukey removed elukey as the assignee of this task.Apr 14 2016, 11:26 AM

Should be fixed now. I did:

root@neodymium:~# salt -v -t 10 -L `cat exmobile` cmd.run 'rm -f /etc/logrotate.d/varnishkafka-*'

(note "exmobile" is just a file listing these ex-mobile cache hosts, which are not in a cluster:cache_foo salt grain at the moment, they're in an odd "temporary" state that has persisted longer than "temporary" should :P)

elukey closed this task as Resolved.Apr 14 2016, 1:07 PM
elukey reopened this task as Open.Apr 19 2016, 6:37 AM

Closed it too soon, I can see the root@ notifications again :(

elukey@cp4003:~$ ls /etc/logrotate.d/varnishkafka*
/etc/logrotate.d/varnishkafka  /etc/logrotate.d/varnishkafka-eventlogging-stats  /etc/logrotate.d/varnishkafka-statsv-stats  /etc/logrotate.d/varnishkafka-webrequest-stats

I've killed the rest of them, I think. I'll let you confirm->close this time :)

Nuria closed this task as Resolved.Apr 27 2016, 4:55 PM