Reaching this task as part of triaging incidents from last year. Release-Engineering-Team would you be able to triage this task?

In T417166#11808469, @MLechvien-WMF wrote:

Reaching this task as part of triaging incidents from last year. Release-Engineering-Team would you be able to triage this task?

Not in its current state. As it says in the description:

We're happy to help once there's a specific request for a change in scap's behavior.

More details about what went wrong in rollback are in the linked Incident document. cc @hnowlan as Incident Coordinator in case you recall anything more specific

This is a challenging action item to make actionable, given the generic problem statement and level of detail available in the linked doc.

The incident itself was a classic (in the "good SRE interview question" sense) metastable failure scenario that prevented rollback of the causative change without additional intervention to make the system either sufficiently healthy (shedding load, killing queries, etc.) or at least appear so to the automation (e.g., disabling health checks, though it does not look like we did that here).

In order to assess whether there's an actionable request for scap here, I'd suggest there are two avenues we can look at ...

Automated supervision - If we had known what to look for, could we have identified impact from the bad change during the canary deployment phase? (Note: I worry that this might be unknowable at this point, given the fact that logs and metrics are no longer available.)

Forced rollback - If scap had the capability to override certain behaviors in a way that would normally be unsafe via an "emergency" rollback mode - e.g., setting maxUnavailable to 100% - is that something we would have felt comfortable reaching for here, and do we (SRE) have a clear definition of what that would look like? (Thought experiment: imagine a world where scap is not involved at all; how would we execute a "health be damned" rollback directly with helmfile and values changes?)

In T417166#11873248, @Scott_French wrote:

Forced rollback - If scap had the capability to override certain behaviors in a way that would normally be unsafe via an "emergency" rollback mode - e.g., setting maxUnavailable to 100% - is that something we would have felt comfortable reaching for here, and do we (SRE) have a clear definition of what that would look like? (Thought experiment: imagine a world where scap is not involved at all; how would we execute a "health be damned" rollback directly with helmfile and values changes?)

Given the description in the task, what @Scott_French outlines here seems like a good first step. That is, scap sets up code, builds containers, and triggers deploys. But the rollouts themselves are heavily dependent on helmfile deploys. If there's an emergency procedure that exists for rollback, then scap can wire it up.