Page MenuHomePhabricator
Create Task
Maniphest T84845

improve cron spam visibility
Open, LowPublic
Actions

Description

right now cron spam is mostly unlooked at, however there might be real issues lurking

I think one of the problems is that there is no aggregation/deduplication so the same issue ends up with many emails and thus ignored, one way to tackle this would be to provide aggregation/deduplication so the real issue is easy to spot (and possibly fix) for example experimenting with https://github.com/KMNR/raven-cron would be a good start. In short cron wouldn't be reporting output via mail but rather via sentry which would then collate based on e.g. the command line

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT294906 Puppet Improvements
 DuplicatejbondT265138 Work required to prepare for puppet 7
 ResolvedSLyngshede-WMFT273673 replace all puppet crons with systemd timers
 OpenNoneT132324 Tracking and Reducing cron-spam to root@
ResolvedelukeyT132322 kafkatee cronspam from oxygen
 ResolvedelukeyT129344 varnishkafka logrotate cronspam
 ResolvedfaidonT84050 Refactor RAID checks (check-raid)
 OpenNoneT83476 Icinga RAID check: monitor rebuild status
 ResolvedfaidonT97998 Add RAID monitoring for HP servers
 ResolvedherronT141252 icinga hp raid check timeout on busy ms-be and db machines
 ResolvedherronT172921 Nrpe command_timeout and "Service Check Timed Out" errors
 ResolvedAndrewT122931 labtestcontrol2001 cronspam
 ResolvedfgiunchediT78762 swift-dispersion-stats cronspam when disks are broken
 DeclinedNoneT1249 Disable cron.standard checking for lost+found directories
 ResolvedjcrespoT127638 Email spam from some MariaDB's logrotate
 ResolvedNoneT132346 cronspam from cpXXXX hosts related to varnishkafka non existent processes
 ResolvedelukeyT132422 cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet
ResolvedNoneT132423 daily cronspam from elastic100* hosts
 ResolvedDzahnT132661 ytterbium, neon and strontium daily cronspam
 Resolved• emaT132835 cronspam from cpXXXX hosts due to update-ocsp-all and zero_fetch
 ResolvedMoritzMuehlenhoffT132896 cronspam from argon - apache2 logrotate
 ResolvedPRODUCTION ERROR• CatropeT137771 "Echo isn't enabled on this wiki" from Cron <www-data@terbium> /usr/local/bin/foreachwiki extensions/Echo/maintenance/processEchoEmailBatch.php >/dev/null
 ResolvedNoneT139943 netmon1001 daily logrotate cronspam
 ResolvedelukeyT144797 graphite-web cronspam
 DuplicateNoneT145360 Cronspam from terbium
 ResolvedNoneT145606 Remove cronspam from stat1002 to root@
 ResolvedNoneT146464 hhvm root:adm owned log files cause failures for logrotate
 ResolvedDzahnT147776 cronspam from cobalt after the Gerrit migration
 ResolvedfaidonT149574 cronspam from labstores, labcontrol, labstestservices
 DuplicateNoneT149681 Cron spam caused by ieee-data cron job
 ResolvedfgiunchediT151149 Prometheus cronspam
 DeclinedNoneT151304 tmpreaper possible race condition
 DeclinedNoneT151314 logrotate failing with $FILE.1.gz: File exists
 DeclinedNoneT151317 stat user crontab on stat hosts for old file removal
 ResolvedNoneT151746 fstrim: Operation not supported on Labs DBs
 ResolvedelukeyT151748 Cron conflict for kafkatee logrotate on oxygen
 OpenNoneT153940 Logrotate fails for: "$FILE No such file or directory"
 ResolvedNoneT156151 Cronspam from mwlog*
 ResolvedfgiunchediT157353 prometheus-vhtcpd-stats cronspamming if vhtcpd is not running yet
 ResolvedJoeT156232 confctl SubjectAltNameWarning after python-urllib3 upgrade
 ResolvedBBlackT159137 certspotter: Error retrieving STH from log
 ResolvedfgiunchediT159437 swift-account-stats: Max retries exceeded with url: /auth/v1.0
 DuplicateNoneT159438 foreachwikiindblist regular cronspam
 ResolvedelukeyT159532 labtestcontrol2001: cron-spam from invoke-rc.d atop _cron
 DeclinedfaidonT162327 certspotter on einsteinium has issues talking to external
 ResolvedMoritzMuehlenhoffT163555 Logrotate fails on mediawiki maintenance servers on jessie
 ResolvedfgiunchediT166937 Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002
 ResolvedcolewhiteT167035 stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist
 ResolvedPRODUCTION ERRORjcrespoT167961 Terbium cronjobs attempting to connect to labstestweb2001
 Resolved• GillesT169249 /usr/local/bin/xenon-generate-svgs and flamegraph.pl cronspam
 Resolved• GillesT187397 Verify latest version of flamegraph.pl locally and submit bug report if needed
 ResolvedfgiunchediT170932 prometheus-puppet-agent-stats cronspam on missing puppet stats
 ResolvedNoneT174959 swift-recon-cron on ms-be203[34]: [Errno 17] File exists: '/var/lock/swift-recon-object-cron'
 DuplicateNoneT176840 nginx xmldumps duplicate logrotate config
 Resolved• chasempT183144 labcontrol1002 Error: unable to connect to node rabbit@labcontrol1002: nodedown
 ResolvedjijikiT150375 cronspam cleanup: Cron <www-data@terbium> /usr/local/bin/foreachwiki maintenance/cleanupUploadStash.php > /dev/null
 ResolvedJoeT185195 tmpreaper doesn't play along with PrivateTmp systemd units
 ResolvedAndrewT192339 labtestweb2001 should either be fully productionized or removed completely from production configuration
 ResolvedjcrespoT192340 Cron <www-data@terbium> /usr/local/bin/mwscriptwikiset extensions/FlaggedRevs/maintenance/updateStats.php flaggedrevs.dblist > /dev/null creates cron spam every 2 hours
Restricted Task
 Resolved• mmodellT196604 Phabricator replica functionality broken, needs port configuration
 ResolvedNoneT197564 cronspam for slow queries in PageAssessments
 DeclinedaaronT216243 cron spam for slow queries on mwmaint /usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null
 Resolved• MarosteguiT216273 New cronspam from db clusters
 ResolvedjbondT217646 wmf-auto-restart occasionally errors on fuse mounts
 Resolved• GTirloniT218987 puppet_alert.py - "It has been 86400 seconds since last Puppet run.Sending nag emails."
 ResolvedelukeyT219111 notebook1004: can't stat() fuse.fuse_dfs file system /mnt/hdfs
 ResolvedMoritzMuehlenhoffT219274 cronspam: cross-validate-accounts
 ResolvedjbondT222443 cron-spam: /usr/local/sbin/check-cumin-aliases
 ResolvedfgiunchediT224399 exim paniclog on $HOST has non-zero size
 ResolvedVolansT224661 cron-spam to root@: lsof stderr generates large emails on boron from wmf-auto-restart
 ResolvedArielGlennT224727 Cron spam from phab1001 delete of temporary files
 ResolvedaaronT228465 TorBlock maintenance failures on labweb hosts
 ResolvedJMeybohmT237197 Kubernetes hosts raid check make facter fail
 Resolved• dpifkeT243762 Daily errors on webperf1002 & webperf2002 /usr/local/bin/arclamp-generate-svgs > /dev/null
 DuplicateNoneT248671 ceph cron-spam emails
 ResolvedcolewhiteT252500 smart-data-dump --syslog producing errors and spamming root@
 Resolved• chasempT254127 peek is incorrectly configured to run every minute every 1st of the month, creating large amounts of cronspam
 DeclinedNoneT257016 Fix paniclog alert to only sent mails once
 ResolvedAndrewT271518 cron spam from cloudcontrol2004-dev.wikimedia.org
Restricted Task
 ResolvedfgiunchediT283951 Logrotate complains about world-writable permissions
 ResolvedBTullisT286442 Trash cleanup cron spams on an-test hosts
 ResolvedJeltoT288324 WARNING: In GitLab 14.0 we will begin removing all configuration backups older than yourgitlab_rails['backup_keep_time'] setting (currently set to: 259200)
 ResolvedVgutierrezT295770 Cronspam from acmechief-test1001
 OpenNoneT311926 PROBLEM: Icinga on alert2001.wikimedia.org is CRITICAL
 ResolvedgregT89049 Make log responsibilities changes
 OpenNoneT106913 Use Sentry on non-production Wikimedia wikis, Toolforge and other sites/tools
 OpenNoneT91724 Set up and test Sentry on Labs for non-JS error logging
DeclinedNoneT106915 Use Sentry in production
 OpenNoneT84845 improve cron spam visibility

Event Timeline

fgiunchedi created this task.Dec 18 2014, 8:46 AM
fgiunchedi raised the priority of this task from to Needs Triage.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added a project: ops-core.
fgiunchedi changed Security from none to None.
fgiunchedi subscribed.
fgiunchedi edited projects, added acl*sre-team; removed ops-core.Dec 18 2014, 9:02 AM
fgiunchedi triaged this task as Low priority.Dec 22 2014, 10:02 AM
• Tgr added a parent task: T91724: Set up and test Sentry on Labs for non-JS error logging .Mar 6 2015, 1:45 AM
• Tgr added a parent task: T106915: Use Sentry in production.Jul 24 2015, 11:45 PM
elukey added a parent task: T132324: Tracking and Reducing cron-spam to root@ .Apr 11 2016, 10:46 AM
fgiunchedi added a comment.Apr 11 2016, 10:52 AM

also note that to test this easily we could either a dummy mailbox to receive cronspam and turn them into sentry messages, this way existing mails wouldn't be touched

Krenair subscribed.Oct 24 2016, 9:11 AM
Dzahn added subscribers: Volans, Dzahn.Oct 18 2017, 3:15 PM

@Volans This seems to be what you mentioned in last monitoring meeting when you suggested an Icinga alert for cronspam.

Volans mentioned this in T178311: Cron spam: figure out a way it doesn't get ignored.Oct 18 2017, 3:19 PM
Volans merged a task: T178311: Cron spam: figure out a way it doesn't get ignored.
Volans added a project: observability.
Volans added a subscriber: faidon.

@Dzahn thanks for pointing this out, I've merged in as duplicate the other task I had opened.

elukey awarded a token.Oct 18 2017, 3:25 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 7:57 PM
colewhite updated the task description. (Show Details)Jun 25 2020, 7:32 PM
colewhite added subscribers: herron, colewhite.Jun 25 2020, 7:41 PM

An option we discussed recently was to ingest mail generated by the servers into Logstash by either pulling events from a mailbox or piping off events at the mail servers. Once in ES, queries could be run and aggregated emails generated as a daily report and/or alerts generated via log alerting.

@herron has asked OIT if a bot inbox is possible.

Ingesting events from a mailbox will also help T230835. If the mx pipe option is chosen, it wouldn't be too difficult to adapt to that paradigm.

fgiunchedi moved this task from Inbox to Backlog on the observability board.Jul 6 2020, 2:12 PM
lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:22 AM
lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:10 AM
lmata edited projects, added Observability-Alerting; removed SRE Observability.Aug 9 2021, 3:17 AM
lmata moved this task from Inbox to Backlog on the Observability-Alerting board.Aug 10 2021, 3:18 PM
lmata moved this task from Backlog to Prioritized on the Observability-Alerting board.Sep 6 2022, 6:45 PM
Dzahn added a comment.EditedSep 6 2022, 7:09 PM

Since this ticket has been written most cron jobs have been converted to systemd timers. Maybe that made all of this obsolete? T273673

Dzahn added a parent task: T273673: replace all puppet crons with systemd timers.Sep 6 2022, 7:14 PM
MoritzMuehlenhoff subscribed.Sep 6 2022, 7:15 PM

cron spam is more of a generic term here, that the spam is now coming from jobs spawned by systemd timers doesn't really change the problem :-)

Dzahn added a comment.Sep 6 2022, 9:26 PM

But the difference is now it's not sent to root@ and instead to actual teams. I would hope that means they don't get ignored any longer which is the original problem statement in this ticket. That being said, I think root mail should also be read by someone and that is a topic actually planned for the SRE summit.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits