Page MenuHomePhabricator

Tracking and Reducing cron-spam to root@
Open, NormalPublic

Description

Cronspam is annoying, but it carries (sometimes) very good information. This phab task should track the efforts to reduce the current level of cronspam from root@ and to keep it under control from now on.

The final goal must be to improve our docs/best-practices/etc.. alongside with the patches that we'll send, otherwise there is no point of proceeding with work.

Some questions:

  1. Do we really value cron-alerts in Wikimedia?
  2. What is an acceptable level of cron-spam?
  3. How do we improve our monitoring/alarming to avoid relying on people reading cron spam emails?

Related Objects

StatusAssignedTask
OpenNone
Resolvedelukey
Resolvedelukey
Resolvedfaidon
OpenNone
Resolvedfaidon
Resolvedherron
Resolvedherron
ResolvedAndrew
Resolvedfgiunchedi
DeclinedNone
OpenNone
Resolvedjcrespo
ResolvedNone
Resolvedelukey
ResolvedNone
Resolved Dzahn
Resolvedema
ResolvedMoritzMuehlenhoff
ResolvedCatrope
ResolvedNone
Resolvedelukey
DuplicateNone
ResolvedNone
ResolvedNone
Resolved Dzahn
Resolvedfaidon
DuplicateNone
Resolvedfgiunchedi
OpenNone
OpenNone
OpenNone
ResolvedNone
Resolvedelukey
OpenNone
ResolvedNone
Resolvedfgiunchedi
OpenNone
ResolvedBBlack
Resolvedfgiunchedi
DuplicateNone
Resolvedelukey
Declinedfaidon
ResolvedMoritzMuehlenhoff
OpenNone
OpenNone
Resolvedjcrespo
ResolvedGilles
ResolvedGilles
Resolvedfgiunchedi
OpenNone
DuplicateNone
Resolvedchasemp
Resolvedjijiki
ResolvedJoe
ResolvedAndrew
Resolvedjcrespo
Resolvedmmodell
OpenJoe
OpenNone
ResolvedMarostegui
Resolvedjbond
ResolvedGTirloni
Resolvedelukey
ResolvedMoritzMuehlenhoff
Resolvedjbond
OpenNone
OpenNone
ResolvedArielGlenn

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 323528 had a related patch set uploaded (by Elukey):
Remove cron notifications to root@ for jobchron/runner service status

https://gerrit.wikimedia.org/r/323528

Change 323528 merged by Faidon Liambotis:
Remove cron notifications to root@ for jobchron/runner service status

https://gerrit.wikimedia.org/r/323528

Mentioned in SAL (#wikimedia-operations) [2016-12-20T08:27:54Z] <elukey> renamed some log files ($something.1.gz to $something.1a.gz) on cp1008 and rutherium to unblock logrotation and reduce cronspam - T132324

Mentioned in SAL (#wikimedia-operations) [2016-12-22T07:26:54Z] <elukey> created /var/log/squid3/access.log.1.gz on aluminum to fix cronspam - T132324

elukey moved this task from Backlog to In Progress on the User-Elukey board.Dec 22 2016, 8:55 AM
elukey moved this task from In Progress to Ops Backlog on the User-Elukey board.Dec 23 2016, 3:12 PM
elukey moved this task from Ops Backlog to Stalled on the User-Elukey board.

Mentioned in SAL (#wikimedia-operations) [2017-01-03T07:58:28Z] <elukey> chown www-data:www-data all the root:adm hhvm log files on mw codfw hosts (T132324)

Mentioned in SAL (#wikimedia-operations) [2017-01-05T07:54:14Z] <elukey> chown www-data:www-data all the root:adm hhvm log files on mw eqiad hosts (T132324)

Change 336218 had a related patch set uploaded (by Elukey):
Silence apt rsync repo activities

https://gerrit.wikimedia.org/r/336218

Change 336218 merged by Dzahn:
Silence apt rsync repo activities

https://gerrit.wikimedia.org/r/336218

Change 343276 had a related patch set uploaded (by Elukey):
[operations/puppet] Add delay compress to upstart's logrotate

https://gerrit.wikimedia.org/r/343276

Change 343276 merged by Elukey:
[operations/puppet] Add delay compress to upstart's logrotate

https://gerrit.wikimedia.org/r/343276

Change 353014 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Fix logrotate config for analytics1003 to avoid cronspam

https://gerrit.wikimedia.org/r/353014

Change 353014 merged by Elukey:
[operations/puppet@production] Fix logrotate config for analytics1003 to avoid cronspam

https://gerrit.wikimedia.org/r/353014

Change 369608 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] statistics::discovery: fix daily logrotate

https://gerrit.wikimedia.org/r/369608

Change 369608 merged by Elukey:
[operations/puppet@production] statistics::discovery: fix daily logrotate

https://gerrit.wikimedia.org/r/369608

Change 370170 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] modules::geowiki::private_data: fix rsync for data-private-bare

https://gerrit.wikimedia.org/r/370170

Change 370170 merged by Elukey:
[operations/puppet@production] modules::geowiki::private_data: fix rsync for data-private-bare

https://gerrit.wikimedia.org/r/370170

EddieGP renamed this task from Tracking and Reducing cron-spam from root@ to Tracking and Reducing cron-spam to root@ .Apr 17 2018, 3:34 PM
ayounsi removed a subscriber: ayounsi.Apr 19 2018, 9:04 PM

Change 428947 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: install SMART checks after 'raid' fact is available.

https://gerrit.wikimedia.org/r/428947

chasemp closed subtask Restricted Task as Resolved.Apr 27 2018, 1:51 PM

Change 438140 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::analytics::refinery::job::sqoop_mw: avoid root cronspam

https://gerrit.wikimedia.org/r/438140

Change 438140 merged by Elukey:
[operations/puppet@production] profile::analytics::refinery::job::sqoop_mw: avoid root cronspam

https://gerrit.wikimedia.org/r/438140