Page MenuHomePhabricator

peek is incorrectly configured to run every minute every 1st of the month, creating large amounts of cronspam
Closed, ResolvedPublic

Event Timeline

jcrespo created this task.Jun 1 2020, 6:31 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 1 2020, 6:31 AM

Change 601170 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] peek: Disable cron execution

https://gerrit.wikimedia.org/r/601170

Change 601170 merged by Jcrespo:
[operations/puppet@production] peek: Disable cron execution

https://gerrit.wikimedia.org/r/601170

Change 601173 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] peek: Reenable cron with the correct configuration

https://gerrit.wikimedia.org/r/601173

jcrespo triaged this task as High priority.EditedJun 1 2020, 6:55 AM

812 reports were sent today 1st June.

The scheduling seems an unintended mixup with cron parameters (* * instead of, e.g. 0 1). It is unclear to me, however, the intention of reporting output, is an email sent by the script and the stdout to root on purpose? Shouldn't the cron be better configured with a systemd timer T210818 and monitor systemd status?

I chose to disable it given it is a report and it could affect Phabricator performance, reenabling can be done at: https://gerrit.wikimedia.org/r/c/operations/puppet/+/601173 but needs Security-Team input (@chasemp).

jcrespo moved this task from Backlog to Radar on the Operations board.Jun 1 2020, 6:56 AM
chasemp added a project: Security-Team.

Thanks for disabling @jcrespo, sorry for the avalanche. Systemd timer seems like a sane idea.

chasemp lowered the priority of this task from High to Medium.Jun 1 2020, 3:06 PM
chasemp edited projects, added Peek; removed Security.Jun 1 2020, 3:09 PM

Change 601173 merged by Rush:
[operations/puppet@production] peek: Reenable cron with correct params

https://gerrit.wikimedia.org/r/601173

sudo -u peek crontab -l
# HEADER: This file was autogenerated at 2020-06-10 16:26:11 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: peek_monthly
MAILTO=security-team@wikimedia.org
0 0 1 * * . $HOME/.profile; /var/lib/peek/git/peek.py -c /etc/peek/config/base.conf,/etc/peek/config/monthly.conf -s > /dev/null
# Puppet Name: peek_weekly
MAILTO=security-team@wikimedia.org
0 0 * * 1 . $HOME/.profile; /var/lib/peek/git/peek.py -c /etc/peek/config/base.conf,/etc/peek/config/weekly.conf -s > /dev/null
chasemp closed this task as Resolved.Jun 10 2020, 4:31 PM