At the moment, mw-cron jobs alert on every failure. In some cases, e.g. in T416440 and T404280, these failures are expected, and do not require human intervention. In these cases, maintenance on the MariaDB servers caused them to be temporarily read-only, and these crons would have succeeded on their next run.
If we expect that regular maintenance may cause transient failures, we should only alert in the circumstances where a human needs to intervene in order to avert operational risk. It would likely be valuable to see if we could instead alert only when a scheduled cron has been unable to successfully complete execution over some duration, rather than alerting on first failed execution.
Some crons may be particularly sensitive, and users may need to be alerted for any failure. This should be configurable (perhaps in Puppet?).
In the past 90 days, there have been 16 instances of "MediaWiki periodic job ... failed" alerts. Of those 16 instances, several will be addressed by the work in T390972, but that work will not cover the type of transient failure seen in T416440 and T404280.