We currently run a nightly health check against the security patches in /srv/patches/next at https://releases-jenkins.wikimedia.org/job/Branch%20cut%20test%20patches/ This will detect any patches that fail to apply cleanly for the code in the master branches and provide instructions on how to fix it for the upcoming (i.e. next) release. These instructions are published on the security patch Phabricator task.

However if a patch is not fixed by the time a new release branch is cut, the patch will make it to the new release at /srv/patches/<version> and prevent that version from being deployed. Right now we have no mechanism in place to report this situation. The scope of this task is to provide such mechanism.

Option 1

We could report the failure only during the weekly train presync on early Tuesdays, but maybe it's simpler and more exhaustive to simply trigger the reporting logic every time there's a patch failure. The current logic already takes care of deciding whether a particular failure has already been reported and avoids spamming.

The SecurityPatchBot's Phabricator token will need to be available to scap on the deployment server in order for notifications to be sent out.

Option 2

The Jenkins nightly also already checks the patches for the current week's release right after a new branch is cut. Similarly to the train presync, this happens every early Tuesday at around 1:20am UTC.

This option is actually simpler and doesn't require adding the bot's Phab token to the deployment server, so I'm leaning in this direction