OCSP:

We have an icinga alert for when things go really-wrong (multiple serial failures), and the commentary in modules/tlsproxy/manifests/ocsp.pp talks about the cron timing vs alert timing and all that:

# Generate icinga alert if OCSP files falling out of date due to errors
#
# Note this makes no provision for un-configured stapling at this time, so
# it will generate warnings if you don't clean up old /var/cache/ocsp/
# entries after removing a tlsproxy::ocsp_stapler cert from a server!
#
# The cron above attempts to get fresh data every hour, and a good fresh
# fetch of data has a 12H lifetime with the windows we're seeing from
# GlobalSign today.
#
# The crit/warn values of 29100 and 14700 correspond are "8h5m" and
# "4h5m", so those are basically warning if 4 updates in a row failed
# for a given cert, and critical if 8 updates in a row failed (at which
# point we have 4h left to fix the situation before the validity window
# expires).

While we're here, it might be wise to make those alert levels a little more-sensitive than they are today for now. I think I was worried about spamming icinga alerts too much during initial implementation, but 4h isn't much notice to fix things on CRIT, either. We should perhaps warn at 3 updates in a row, and crit at 5 updates in a row, leaving us a ~7h window to fix the CRIT? Will upload a trivial patch for that.

The only other reason (aside from alerting) to have the cronspam on failure is that we don't otherwise have a log to look back through on why/how each failure happened if we're debugging icinga alerts about the files. We could perhaps add some trivial logging to the current cron (for all output of update-ocsp-all, appending to e.g. /var/log/update-ocsp-all, and with a logrotate config to keep only a week?)

Note also T93927 about general improvements to our OCSP infra. Not worked on in a long time because it hasn't been a pragmatic issue in a long time.

Zerofetcher:

Similar in nature to OCSP, in that it's a cron script that updates files on disk periodically. The notable differences are these:

• The source of the updates is in-house, so there should be far fewer failures and we can actually debug the failure causes better.
• Actual changes to the data are ultimately triggered by human administrative action (someone on the Zero team edits some JSON in an internal wiki, zerofetcher syncs down that JSON to all the caches).
• We have no icinga alert on the freshness (in mtime terms) of the output files, because by design the actual output files are untouched if there was no upstream data change (because the varnish vmod that re-loads the data is mtime-sensitive, so not touching the files on no-op avoids pointless reloads). We could perhaps fix this by having the script touch a different file on every successful run and icinga-alerting on the freshness of that. The baseline timing is every 15 minutes and it's not quite so critical if this goes stale (in operational terms!), so perhaps warn if it goes stale for 4 hours, CRIT if it goes stale for a day?
• And then once again, we have no built-in logging for this, so the cron output is also effectively a log channel. Adding rotated local logging to it would address that.

TL;DR - TODO List:

1. Make OCSP icinga alert more aggressive (trivial patch inc right after this reply)
2. Modify update-ocsp-all script/cron to log all output to /var/log/update-ocsp-all.log (preventing cronspam output to stdio), and add logrotate script for it.
3. Modify the zerofetch.py script to touch some file on any successful run, perhaps /var/cache/ocsp/.update-success.
4. Add an icinga alert on stale-ness of that file (OCSP uses an NRPE script that would work, if a little inefficiently)
5. As with OCSP, send the cron output to some local logfile and rotate it.

Change 283767 had a related patch set uploaded (by BBlack):
OCSP Stapling: make icinga alerts more aggressive

https://gerrit.wikimedia.org/r/283767

Change 283767 merged by BBlack:
OCSP Stapling: make icinga alerts more aggressive

https://gerrit.wikimedia.org/r/283767

Change 291752 had a related patch set uploaded (by Ema):
update-ocsp-all: write output to logfile

https://gerrit.wikimedia.org/r/291752

Change 291752 merged by Ema:
update-ocsp-all: write output to logfile

https://gerrit.wikimedia.org/r/291752

Change 294063 had a related patch set uploaded (by Ema):
zerofetch.py: track successful executions

https://gerrit.wikimedia.org/r/294063

Change 294072 had a related patch set uploaded (by Ema):
zerofetch icinga check

https://gerrit.wikimedia.org/r/294072

Change 294077 had a related patch set uploaded (by Ema):
zerofetch: write output to logfile

https://gerrit.wikimedia.org/r/294077

Change 294063 merged by Ema:
zerofetch.py: track successful executions

https://gerrit.wikimedia.org/r/294063

Change 294072 merged by Ema:
zerofetch icinga check

https://gerrit.wikimedia.org/r/294072

Change 294077 merged by Ema:
zerofetch: write output to logfile

https://gerrit.wikimedia.org/r/294077