@Legoktm is working on a cookbook to speed up packaging of scap https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/727605. The rollout process has to stay as it is though (upgrade on canaries first, and roll out to all hosts after 1-2 days)

In T292646#7419245, @jijiki wrote:

The rollout process has to stay as it is though (upgrade on canaries first, and roll out to all hosts after 1-2 days)

For the initial version I'm just tackling the packaging part, but after that we could add the debdeploy to canaries and even some test syncs from the deployment server.

I'm curious what the goal of the 1-2 day delay is. In theory if we can release scap faster, then we can also fix potential regressions faster so catching them at the canary stage is less important. Obviously that depends on what kind of regressions we're expecting, which is what I'm curious about - which code paths or uses of scap need 1-2 days to test/find issues with, and is there a way we could test those things during the scap release/deploy process?

@Legoktm we may debdeploy scap everywhere, and then for whatever reason we need to push change Y fast due to issue X. If scap fails everywhere because of a bug we missed, we have a problem where we first need to downgrade scap, and then rerun it. In my opinion, we should keep having scap sit on the canaries for 1 day, and save us from a potential scenario like this. To my knowledge, scap's test coverage is rather low (I admit I have not read scap code for quite some time). If this is still the case, gives us one more reason to want to be a little bit more careful with its rollout.