Page MenuHomePhabricator
Create Task
Maniphest T132324

Tracking and Reducing cron-spam to root@
Open, MediumPublic
Actions

Description

Cronspam is annoying, but it carries (sometimes) very good information. This phab task should track the efforts to reduce the current level of cronspam from root@ and to keep it under control from now on.

The final goal must be to improve our docs/best-practices/etc.. alongside with the patches that we'll send, otherwise there is no point of proceeding with work.

Some questions:

  1. Do we really value cron-alerts in Wikimedia?
  2. What is an acceptable level of cron-spam?
  3. How do we improve our monitoring/alarming to avoid relying on people reading cron spam emails?

Details

SubjectRepoBranchLines +/-
profile: install SMART checks after 'raid' fact is available.operations/puppetproduction+1 -1
swap: Redirect stderr to /dev/null to prevent cronspamoperations/puppetproduction+1 -1
profile::analytics::refinery::job::sqoop_mw: avoid root cronspamoperations/puppetproduction+12 -10
modules::geowiki::private_data: fix rsync for data-private-bareoperations/puppetproduction+1 -1
statistics::discovery: fix daily logrotateoperations/puppetproduction+1 -0
Fix logrotate config for analytics1003 to avoid cronspamoperations/puppetproduction+1 -0
Add delay compress to upstart's logrotateoperations/puppetproduction+1 -0
Silence apt rsync repo activitiesoperations/puppetproduction+1 -1
Remove cron notifications to root@ for jobchron/runner service statusoperations/puppetproduction+2 -2
Set l10nupdate group in its related logrotate config fileoperations/puppetproduction+4 -5
l10nupdate: Add 'su' to logrotate scriptoperations/puppetproduction+3 -2
Add the logrotate delaycompress setting everywhereoperations/puppetproduction+35 -1
Reduce cronspam to root@ from zookeeper zkCleanup scriptoperations/puppetproduction+11 -1
Attempt to stop cronspam from graphite-weboperations/puppetproduction+2 -3
Remove g+w flag to /srv/log/eventlogging to allow proper logrotationoperations/puppetproduction+1 -1
Reduce cronspam from terbium related to the echo_mail_batch cron scriptoperations/puppetproduction+2 -2
Remove cronspam coming from Gerrit log deletionoperations/puppetproduction+1 -1
mediawiki: add delaycompress for jobrunner logsoperations/puppetproduction+1 -0
Avoid log spam as 99% of these would be non-errorsoperations/puppetproduction+1 -1
Don't install apt-show-versionsoperations/puppetproduction+0 -20
update-ocsp-all: write output to logfileoperations/puppetproduction+20 -1
Add mod_headers to the httpd config to allow the Header directive.operations/puppetproduction+1 -0
Add delaycompress to ganglia-web's logrotate to avoid daily cronspam.operations/puppetproduction+1 -0
Update the kafkatee submodule with the latest rsyslog/logrotate updates.operations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT294906 Puppet Improvements
 DuplicatejbondT265138 Work required to prepare for puppet 7
 ResolvedSLyngshede-WMFT273673 replace all puppet crons with systemd timers
 OpenNoneT132324 Tracking and Reducing cron-spam to root@
ResolvedelukeyT132322 kafkatee cronspam from oxygen
 ResolvedelukeyT129344 varnishkafka logrotate cronspam
 ResolvedfaidonT84050 Refactor RAID checks (check-raid)
 OpenNoneT83476 Icinga RAID check: monitor rebuild status
 ResolvedfaidonT97998 Add RAID monitoring for HP servers
 ResolvedherronT141252 icinga hp raid check timeout on busy ms-be and db machines
 ResolvedherronT172921 Nrpe command_timeout and "Service Check Timed Out" errors
 ResolvedAndrewT122931 labtestcontrol2001 cronspam
 ResolvedfgiunchediT78762 swift-dispersion-stats cronspam when disks are broken
 DeclinedNoneT1249 Disable cron.standard checking for lost+found directories
 OpenNoneT84845 improve cron spam visibility
 ResolvedjcrespoT127638 Email spam from some MariaDB's logrotate
 ResolvedNoneT132346 cronspam from cpXXXX hosts related to varnishkafka non existent processes
 ResolvedelukeyT132422 cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet
ResolvedNoneT132423 daily cronspam from elastic100* hosts
 ResolvedDzahnT132661 ytterbium, neon and strontium daily cronspam
 Resolved• emaT132835 cronspam from cpXXXX hosts due to update-ocsp-all and zero_fetch
 ResolvedMoritzMuehlenhoffT132896 cronspam from argon - apache2 logrotate
 ResolvedPRODUCTION ERRORCatropeT137771 "Echo isn't enabled on this wiki" from Cron <www-data@terbium> /usr/local/bin/foreachwiki extensions/Echo/maintenance/processEchoEmailBatch.php >/dev/null
 ResolvedNoneT139943 netmon1001 daily logrotate cronspam
 ResolvedelukeyT144797 graphite-web cronspam
 DuplicateNoneT145360 Cronspam from terbium
 ResolvedNoneT145606 Remove cronspam from stat1002 to root@
 ResolvedNoneT146464 hhvm root:adm owned log files cause failures for logrotate
 ResolvedDzahnT147776 cronspam from cobalt after the Gerrit migration
 ResolvedfaidonT149574 cronspam from labstores, labcontrol, labstestservices
 DuplicateNoneT149681 Cron spam caused by ieee-data cron job
 ResolvedfgiunchediT151149 Prometheus cronspam
 DeclinedNoneT151304 tmpreaper possible race condition
 DeclinedNoneT151314 logrotate failing with $FILE.1.gz: File exists
 DeclinedNoneT151317 stat user crontab on stat hosts for old file removal
 ResolvedNoneT151746 fstrim: Operation not supported on Labs DBs
 ResolvedelukeyT151748 Cron conflict for kafkatee logrotate on oxygen
 OpenNoneT153940 Logrotate fails for: "$FILE No such file or directory"
 ResolvedNoneT156151 Cronspam from mwlog*
 ResolvedfgiunchediT157353 prometheus-vhtcpd-stats cronspamming if vhtcpd is not running yet
 ResolvedJoeT156232 confctl SubjectAltNameWarning after python-urllib3 upgrade
 ResolvedBBlackT159137 certspotter: Error retrieving STH from log
 ResolvedfgiunchediT159437 swift-account-stats: Max retries exceeded with url: /auth/v1.0
 DuplicateNoneT159438 foreachwikiindblist regular cronspam
 ResolvedelukeyT159532 labtestcontrol2001: cron-spam from invoke-rc.d atop _cron
 DeclinedfaidonT162327 certspotter on einsteinium has issues talking to external
 ResolvedMoritzMuehlenhoffT163555 Logrotate fails on mediawiki maintenance servers on jessie
 ResolvedfgiunchediT166937 Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002
 ResolvedcolewhiteT167035 stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist
 ResolvedPRODUCTION ERRORjcrespoT167961 Terbium cronjobs attempting to connect to labstestweb2001
 Resolved• GillesT169249 /usr/local/bin/xenon-generate-svgs and flamegraph.pl cronspam
 Resolved• GillesT187397 Verify latest version of flamegraph.pl locally and submit bug report if needed
 ResolvedfgiunchediT170932 prometheus-puppet-agent-stats cronspam on missing puppet stats
 ResolvedNoneT174959 swift-recon-cron on ms-be203[34]: [Errno 17] File exists: '/var/lock/swift-recon-object-cron'
 DuplicateNoneT176840 nginx xmldumps duplicate logrotate config
 Resolved• chasempT183144 labcontrol1002 Error: unable to connect to node rabbit@labcontrol1002: nodedown
 ResolvedjijikiT150375 cronspam cleanup: Cron <www-data@terbium> /usr/local/bin/foreachwiki maintenance/cleanupUploadStash.php > /dev/null
 ResolvedJoeT185195 tmpreaper doesn't play along with PrivateTmp systemd units
 ResolvedAndrewT192339 labtestweb2001 should either be fully productionized or removed completely from production configuration
 ResolvedjcrespoT192340 Cron <www-data@terbium> /usr/local/bin/mwscriptwikiset extensions/FlaggedRevs/maintenance/updateStats.php flaggedrevs.dblist > /dev/null creates cron spam every 2 hours
Restricted Task
 Resolved• mmodellT196604 Phabricator replica functionality broken, needs port configuration
 ResolvedNoneT197564 cronspam for slow queries in PageAssessments
 DeclinedaaronT216243 cron spam for slow queries on mwmaint /usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null
 Resolved• MarosteguiT216273 New cronspam from db clusters
 ResolvedjbondT217646 wmf-auto-restart occasionally errors on fuse mounts
 Resolved• GTirloniT218987 puppet_alert.py - "It has been 86400 seconds since last Puppet run.Sending nag emails."
 ResolvedelukeyT219111 notebook1004: can't stat() fuse.fuse_dfs file system /mnt/hdfs
 ResolvedMoritzMuehlenhoffT219274 cronspam: cross-validate-accounts
 ResolvedjbondT222443 cron-spam: /usr/local/sbin/check-cumin-aliases
 ResolvedfgiunchediT224399 exim paniclog on $HOST has non-zero size
 ResolvedVolansT224661 cron-spam to root@: lsof stderr generates large emails on boron from wmf-auto-restart
 ResolvedArielGlennT224727 Cron spam from phab1001 delete of temporary files
 ResolvedaaronT228465 TorBlock maintenance failures on labweb hosts
 ResolvedJMeybohmT237197 Kubernetes hosts raid check make facter fail
 Resolved• dpifkeT243762 Daily errors on webperf1002 & webperf2002 /usr/local/bin/arclamp-generate-svgs > /dev/null
 DuplicateNoneT248671 ceph cron-spam emails
 ResolvedcolewhiteT252500 smart-data-dump --syslog producing errors and spamming root@
 Resolved• chasempT254127 peek is incorrectly configured to run every minute every 1st of the month, creating large amounts of cronspam
 OpenNoneT257016 Fix paniclog alert to only sent mails once
 ResolvedAndrewT271518 cron spam from cloudcontrol2004-dev.wikimedia.org
Restricted Task
 ResolvedfgiunchediT283951 Logrotate complains about world-writable permissions
 ResolvedBTullisT286442 Trash cleanup cron spams on an-test hosts
 ResolvedJeltoT288324 WARNING: In GitLab 14.0 we will begin removing all configuration backups older than yourgitlab_rails['backup_keep_time'] setting (currently set to: 259200)
 ResolvedVgutierrezT295770 Cronspam from acmechief-test1001
 OpenNoneT311926 PROBLEM: Icinga on alert2001.wikimedia.org is CRITICAL

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
elukey removed a project: User-Elukey.May 2 2017, 4:25 PM
gerritbot added a comment.May 10 2017, 7:46 AM

Change 353014 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Fix logrotate config for analytics1003 to avoid cronspam

https://gerrit.wikimedia.org/r/353014

gerritbot added a comment.May 10 2017, 7:48 AM

Change 353014 merged by Elukey:
[operations/puppet@production] Fix logrotate config for analytics1003 to avoid cronspam

https://gerrit.wikimedia.org/r/353014

elukey created subtask T166937: Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002.Jun 3 2017, 6:12 AM
fgiunchedi created subtask T167035: stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist.Jun 5 2017, 3:54 PM
elukey closed subtask T146464: hhvm root:adm owned log files cause failures for logrotate as Resolved.Jun 12 2017, 4:01 PM
jcrespo added a subtask: T167961: Terbium cronjobs attempting to connect to labstestweb2001.Jun 15 2017, 3:17 PM
• Marostegui closed subtask T167961: Terbium cronjobs attempting to connect to labstestweb2001 as Resolved.Jun 19 2017, 2:59 PM
fgiunchedi created subtask T169249: /usr/local/bin/xenon-generate-svgs and flamegraph.pl cronspam.Jun 29 2017, 5:03 PM
elukey closed subtask T151748: Cron conflict for kafkatee logrotate on oxygen as Resolved.Jul 3 2017, 8:01 AM
fgiunchedi created subtask T170932: prometheus-puppet-agent-stats cronspam on missing puppet stats.Jul 18 2017, 1:39 PM
fgiunchedi closed subtask T157353: prometheus-vhtcpd-stats cronspamming if vhtcpd is not running yet as Resolved.Jul 19 2017, 2:37 PM
faidon closed subtask T162327: certspotter on einsteinium has issues talking to external as Declined.Jul 24 2017, 3:39 PM
fgiunchedi closed subtask T170932: prometheus-puppet-agent-stats cronspam on missing puppet stats as Resolved.Jul 27 2017, 2:21 PM
gerritbot added a comment.Aug 2 2017, 8:20 AM

Change 369608 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] statistics::discovery: fix daily logrotate

https://gerrit.wikimedia.org/r/369608

gerritbot added a comment.Aug 2 2017, 8:25 AM

Change 369608 merged by Elukey:
[operations/puppet@production] statistics::discovery: fix daily logrotate

https://gerrit.wikimedia.org/r/369608

gerritbot added a comment.Aug 4 2017, 9:19 AM

Change 370170 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] modules::geowiki::private_data: fix rsync for data-private-bare

https://gerrit.wikimedia.org/r/370170

gerritbot added a comment.Aug 4 2017, 9:22 AM

Change 370170 merged by Elukey:
[operations/puppet@production] modules::geowiki::private_data: fix rsync for data-private-bare

https://gerrit.wikimedia.org/r/370170

Krinkle closed subtask T1249: Disable cron.standard checking for lost+found directories as Declined.Aug 8 2017, 2:11 AM
• ema created subtask T174959: swift-recon-cron on ms-be203[34]: [Errno 17] File exists: '/var/lock/swift-recon-object-cron'.Sep 4 2017, 4:09 PM
elukey created subtask T176840: nginx xmldumps duplicate logrotate config.Sep 27 2017, 8:43 AM
BBlack closed subtask T159137: certspotter: Error retrieving STH from log as Resolved.Oct 23 2017, 3:51 PM
Volans added a subtask: T150375: cronspam cleanup: Cron <www-data@terbium> /usr/local/bin/foreachwiki maintenance/cleanupUploadStash.php > /dev/null.Dec 18 2017, 2:26 PM
• chasemp closed subtask T183144: labcontrol1002 Error: unable to connect to node rabbit@labcontrol1002: nodedown as Resolved.Jan 2 2018, 2:39 PM
elukey added a subtask: T185195: tmpreaper doesn't play along with PrivateTmp systemd units.Jan 18 2018, 10:17 AM
EddieGP renamed this task from Tracking and Reducing cron-spam from root@ to Tracking and Reducing cron-spam to root@ .Apr 17 2018, 3:34 PM
Andrew closed subtask T192339: labtestweb2001 should either be fully productionized or removed completely from production configuration as Resolved.Apr 19 2018, 5:42 PM
ayounsi unsubscribed.Apr 19 2018, 9:04 PM
• Gilles closed subtask T169249: /usr/local/bin/xenon-generate-svgs and flamegraph.pl cronspam as Resolved.Apr 24 2018, 7:40 AM
jcrespo closed subtask T192340: Cron <www-data@terbium> /usr/local/bin/mwscriptwikiset extensions/FlaggedRevs/maintenance/updateStats.php flaggedrevs.dblist > /dev/null creates cron spam every 2 hours as Resolved.Apr 25 2018, 3:59 PM
gerritbot added a comment.Apr 25 2018, 4:19 PM

Change 428947 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: install SMART checks after 'raid' fact is available.

https://gerrit.wikimedia.org/r/428947

• chasemp closed subtask Restricted Task as Resolved.Apr 27 2018, 1:51 PM
jcrespo added a subtask: T196604: Phabricator replica functionality broken, needs port configuration.Jun 7 2018, 1:06 PM
gerritbot added a comment.Jun 8 2018, 6:22 AM

Change 438140 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::analytics::refinery::job::sqoop_mw: avoid root cronspam

https://gerrit.wikimedia.org/r/438140

gerritbot added a comment.Jun 8 2018, 6:23 AM

Change 438140 merged by Elukey:
[operations/puppet@production] profile::analytics::refinery::job::sqoop_mw: avoid root cronspam

https://gerrit.wikimedia.org/r/438140

• mmodell closed subtask T196604: Phabricator replica functionality broken, needs port configuration as Resolved.Jun 8 2018, 8:13 PM
• Vvjjkkii reopened subtask T196604: Phabricator replica functionality broken, needs port configuration as Open.Jul 1 2018, 1:06 AM
• Marostegui closed subtask T196604: Phabricator replica functionality broken, needs port configuration as Resolved.Jul 2 2018, 5:04 AM
jijiki closed subtask T150375: cronspam cleanup: Cron <www-data@terbium> /usr/local/bin/foreachwiki maintenance/cleanupUploadStash.php > /dev/null as Resolved.Jan 8 2019, 12:08 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:36 PM
Joe closed subtask T185195: tmpreaper doesn't play along with PrivateTmp systemd units as Resolved.Feb 12 2019, 6:58 AM
jcrespo added a subtask: T197564: cronspam for slow queries in PageAssessments.Feb 15 2019, 1:46 PM
jcrespo added a subtask: T216243: cron spam for slow queries on mwmaint /usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null.Feb 15 2019, 2:04 PM
colewhite added a subtask: T216273: New cronspam from db clusters.Feb 15 2019, 7:32 PM
• Marostegui closed subtask T216273: New cronspam from db clusters as Resolved.Feb 19 2019, 2:09 PM
jbond closed subtask T217646: wmf-auto-restart occasionally errors on fuse mounts as Resolved.Mar 15 2019, 10:51 AM
jcrespo added a subtask: T218987: puppet_alert.py - "It has been 86400 seconds since last Puppet run.Sending nag emails.".Mar 22 2019, 12:25 PM
jcrespo mentioned this in T218987: puppet_alert.py - "It has been 86400 seconds since last Puppet run.Sending nag emails.".
• GTirloni added a subtask: T219111: notebook1004: can't stat() fuse.fuse_dfs file system /mnt/hdfs.Mar 25 2019, 2:09 AM
elukey closed subtask T219111: notebook1004: can't stat() fuse.fuse_dfs file system /mnt/hdfs as Resolved.Mar 25 2019, 4:18 PM
MoritzMuehlenhoff closed subtask T219274: cronspam: cross-validate-accounts as Resolved.Apr 9 2019, 10:48 AM
bd808 closed subtask T218987: puppet_alert.py - "It has been 86400 seconds since last Puppet run.Sending nag emails." as Resolved.Jun 3 2019, 2:38 AM
ArielGlenn closed subtask T224727: Cron spam from phab1001 delete of temporary files as Resolved.Jun 10 2019, 7:05 AM
jbond closed subtask T222443: cron-spam: /usr/local/sbin/check-cumin-aliases as Resolved.Jun 19 2019, 2:29 PM
Krinkle closed subtask T216243: cron spam for slow queries on mwmaint /usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null as Declined.Jul 2 2019, 2:44 PM
jcrespo added a subtask: T228465: TorBlock maintenance failures on labweb hosts.Jul 19 2019, 8:27 AM
tstarling closed subtask T228465: TorBlock maintenance failures on labweb hosts as Resolved.Jul 22 2019, 4:51 AM
jijiki closed subtask T151304: tmpreaper possible race condition as Resolved.Oct 8 2019, 11:33 AM
MoritzMuehlenhoff reopened subtask T151304: tmpreaper possible race condition as Open.Oct 8 2019, 11:54 AM
gerritbot added a comment.Oct 17 2019, 2:44 PM

Change 543866 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Redirect stderr to /dev/null to prevent cronspam

https://gerrit.wikimedia.org/r/543866

gerritbot added a comment.Oct 22 2019, 7:40 AM

Change 543866 merged by Elukey:
[operations/puppet@production] swap: Redirect stderr to /dev/null to prevent cronspam

https://gerrit.wikimedia.org/r/543866

bd808 closed subtask T151304: tmpreaper possible race condition as Declined.Jan 1 2020, 6:14 PM
Krinkle closed subtask T243762: Daily errors on webperf1002 & webperf2002 /usr/local/bin/arclamp-generate-svgs > /dev/null as Resolved.Jan 29 2020, 7:41 PM
fgiunchedi closed subtask T166937: Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002 as Resolved.Apr 14 2020, 3:46 PM
TerraCodes unsubscribed.Apr 15 2020, 12:09 PM
colewhite closed subtask T167035: stretch acct monthly cron will spam when /var/log/wtmp.1 doesn't exist as Resolved.May 5 2020, 4:38 PM
• Marostegui closed subtask T224661: cron-spam to root@: lsof stderr generates large emails on boron from wmf-auto-restart as Resolved.May 12 2020, 6:34 AM
• Marostegui added a subtask: T252500: smart-data-dump --syslog producing errors and spamming root@.May 12 2020, 6:40 AM
• Marostegui closed subtask T197564: cronspam for slow queries in PageAssessments as Resolved.May 12 2020, 6:43 AM
• Marostegui closed subtask T174959: swift-recon-cron on ms-be203[34]: [Errno 17] File exists: '/var/lock/swift-recon-object-cron' as Resolved.May 12 2020, 6:46 AM
colewhite closed subtask T252500: smart-data-dump --syslog producing errors and spamming root@ as Resolved.May 12 2020, 4:15 PM
jcrespo added a subtask: T254127: peek is incorrectly configured to run every minute every 1st of the month, creating large amounts of cronspam.Jun 1 2020, 6:33 AM
• chasemp closed subtask T254127: peek is incorrectly configured to run every minute every 1st of the month, creating large amounts of cronspam as Resolved.Jun 10 2020, 4:31 PM
MoritzMuehlenhoff added a subtask: T257016: Fix paniclog alert to only sent mails once.Jul 3 2020, 6:34 AM
fgiunchedi closed subtask T224399: exim paniclog on $HOST has non-zero size as Resolved.Jul 6 2020, 11:31 AM
gerritbot added a comment.Jul 31 2020, 12:31 PM

Change 428947 abandoned by Filippo Giunchedi:
[operations/puppet@production] profile: install SMART checks after 'raid' fact is available.

Reason:
Not an issue

https://gerrit.wikimedia.org/r/428947

jijiki reopened subtask T252500: smart-data-dump --syslog producing errors and spamming root@ as Open.Aug 26 2020, 3:10 PM
jijiki closed subtask T252500: smart-data-dump --syslog producing errors and spamming root@ as Resolved.Aug 26 2020, 3:13 PM
taavi added a project: Tracking-Neverending.Jan 8 2021, 10:18 AM
jcrespo added a subtask: Restricted Task.Jan 20 2021, 11:51 AM
jcrespo closed subtask Restricted Task as Resolved.Jan 25 2021, 10:02 AM
JMeybohm closed subtask T237197: Kubernetes hosts raid check make facter fail as Resolved.Feb 5 2021, 8:42 AM
jcrespo added a subtask: T283951: Logrotate complains about world-writable permissions.May 31 2021, 3:25 PM
fgiunchedi closed subtask T283951: Logrotate complains about world-writable permissions as Resolved.Jun 1 2021, 9:36 AM
Aklapper removed a project: Patch-For-Review.Jun 6 2021, 9:29 PM
LSobanski subscribed.Aug 9 2021, 9:11 AM
Jelto closed subtask T288324: WARNING: In GitLab 14.0 we will begin removing all configuration backups older than yourgitlab_rails['backup_keep_time'] setting (currently set to: 259200) as Resolved.Sep 15 2021, 6:41 AM
taavi closed subtask T271518: cron spam from cloudcontrol2004-dev.wikimedia.org as Resolved.Nov 7 2021, 5:33 PM
Vgutierrez closed subtask T295770: Cronspam from acmechief-test1001 as Resolved.Nov 16 2021, 2:39 PM
Jelto reopened subtask T295770: Cronspam from acmechief-test1001 as Open.Nov 17 2021, 9:34 AM
gerritbot added a comment.Dec 1 2021, 11:06 PM

Change 743040 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] exim: Reenable regular flash sale offers on wmf address

https://gerrit.wikimedia.org/r/743040

gerritbot added a project: Patch-For-Review.Dec 1 2021, 11:06 PM
Dzahn added a parent task: T273673: replace all puppet crons with systemd timers.Feb 11 2022, 7:18 PM
Joe closed subtask T156232: confctl SubjectAltNameWarning after python-urllib3 upgrade as Resolved.Mar 2 2022, 7:15 AM
fgiunchedi closed subtask T151314: logrotate failing with $FILE.1.gz: File exists as Declined.Apr 29 2022, 12:49 PM
fgiunchedi closed subtask T151317: stat user crontab on stat hosts for old file removal as Declined.Apr 29 2022, 1:01 PM
jcrespo added a subtask: T311926: PROBLEM: Icinga on alert2001.wikimedia.org is CRITICAL.Jul 2 2022, 5:08 PM
BTullis closed subtask T286442: Trash cleanup cron spams on an-test hosts as Resolved.Jun 13 2023, 1:51 PM
Vgutierrez closed subtask T295770: Cronspam from acmechief-test1001 as Resolved.Jul 10 2023, 11:58 AM
jhathaway subscribed.Aug 21 2023, 1:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL