I found one very likely explanation, b7b54659f089491 was deployed on Feb 19 and kills scripts after 1 hour. This means that we're probably running the BetaFeatures script for 1 hour every day, and killing it!

Potential steps forward:

• Add code to log and report out when a script times out.
• Raise the time limit.
• Rewrite the BetaFeatures report to finish within the timeout.

In T260176#6399916, @awight wrote:

• Add code to log and report out when a script times out.

It logs error AFAIK. It's in stat1007, I can try to find it for you but I'm sure it logs them.

Thanks, it looks like we have many misbehaving scripts (extra linefeeds are my own):

2020-08-17 03:17:00 wikidata-datamodel-statements_per_entity Script Finished!
sh: 1: Syntax error: EOF in backquote substitution

2020-08-17 03:54:32 wikidata-entityUsage Script Finished!

PHP Fatal error:  Maximum execution time of 3600 seconds exceeded in /srv/analytics-wmde/graphite/src/scripts/lib/WikimediaDb.php on line 68

2020-08-18 03:01:48 wikidata-dumpScanProcessing File not found: /srv/analytics-wmde/graphite/data/20190729/metrics.json
2020-08-18 03:01:48 wikidata-dumpScanProcessing File not found: /srv/analytics-wmde/graphite/data/20190722/metrics.json
2020-08-18 03:01:48 wikidata-dumpScanProcessing File not found: /srv/analytics-wmde/graphite/data/20190717/metrics.json

PHP Warning:  Invalid argument supplied for foreach() in /srv/analytics-wmde/graphite/src/scripts/src/wikidata/sparql/ranks.php on line 37
2020-08-18 03:01:01 wikidata-sparql-ranks Script Finished!

2020-08-17 12:00:01 wikidata-apiLogScanner Script Started!
PHP Fatal error:  Uncaught RuntimeException: File does not exist: /srv/log/mw-log/archive/api/api.log-20200815.gz in /srv/analytics-wmde/graphite/src/scripts/src/wikidata/apiLogScanner.php:105
Stack trace:
#0 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/apiLogScanner.php(52): WikidataApiLogScanner->dieIfFilesDoNotExist()
#1 /srv/analytics-wmde/graphite/src/scripts/src/wikidata/apiLogScanner.php(22): WikidataApiLogScanner->execute()
#2 {main}
  thrown in /srv/analytics-wmde/graphite/src/scripts/src/wikidata/apiLogScanner.php on line 105

It looks like the timeout happens at a random time, out of sync with the surrounding loglines. This makes sense because all jobs are started as background tasks from the shell, so they may be logging concurrently. The shell has no information about which job expired. It's possible that the first timing-out job kills any other long-running jobs. Indeed, I can see that the "betafeature-counts Script Started!" loglines never have a corresponding "Script Finished".

Matching these lines up over the last few days:

All of these scripts are failing to complete, for unknown reasons:

advancedsearch-userprops
betafeature-counts
catwatch-userprops
echo-statusNotifications
revslider-userprops
rollbackconf-userprops
twocolconflict-userprops

I suggest that we get rid of the "daily.*.sh" scripts and instead call each script directly from cron. We could still share a common wrapper around each command, but listing them separately in cron gives built-in visibility about which scripts fail.

I suggest that we get rid of the "daily.*.sh" scripts and instead call each script directly from cron. We could still share a common wrapper around each command, but listing them separately in cron gives built-in visibility about which scripts fail.

Sounds good as these scripts etc are indeed getting more complicated.
The reason for a single cron file in the first place was so that we could avoid touching puppet a bunch.
Many of these scripts could likely use data from other faster sources now (like hadoop)
It sould like we should also do an audit of these scripts and see if we can kill some of them (especially any long failing ones)