A known problem with both the grid engine cron and the new jobs framework is thundering herd problems for the execution environment schedulers at popular times (00:00, *:00, *:30, etc).
Because TJF is acting as an intermediary between the user and the Kubernetes backend we have an opportunity to introduce support for a system generated skew or offset for users attempting to avoid the thundering herd. As an intermediary, the skew that is introduced cannot easily be a dynamic response of the system. We do however have an opportunity to translate the schedule provided by the user into a slightly different set of fixed values when passing data to the Kubernetes API.
Some ideas:
- Translate @yearly, @annually, @monthly, @weekly, @daily, @midnight, and @hourly into specific values with randomized offsets from hour & minute zero as appropriate to the period.
- Take inspiration from Drupal's Ultimate Cron and add support for their "system generated offset" (+@) concept. See CronRule:preparePart(), UltimateCronCrontabScheduler:getSkew(), and UltimateCronJob:getUniqueID()
- Skew all jobs by default and provide syntax to opt-out of that (likely controversial)