Add failure rate triggered rollback to scap
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Clement_Goubert
	Sep 9 2022, 10:45 AM

Description

scap currently only stops if a canary hosts fail deployment.

I suggest that, further, when a certain percentage of failure to restart or repool is being reported by scap for a cluster after canaries pass, it should:

either stop or rollback
instruct on contact information for checking failure cause

Percentage of failure tbd.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T317340 Incident: 2022-09-08 codfw appservers degradation
		Open		None	T317405 Add failure rate triggered rollback to scap

Event Timeline

Clement_Goubert created this task.Sep 9 2022, 10:45 AM

Clement_Goubert updated the task description. (Show Details)Sep 9 2022, 3:44 PM

jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:17 PM

Volans added a project: SRE-Sprint-Week-Sustainability-March2023.Mar 20 2023, 3:44 PM

Volans moved this task from Backlog to E_TOO_BIG_MAYBE_OKR? on the SRE-Sprint-Week-Sustainability-March2023 board.

Re-tagging to the team responsible for scap.

dancy mentioned this in T239376: Run Swagger checks in Scap before exposing to prod MW traffic .Feb 15 2024, 9:27 PM

Add failure rate triggered rollback to scapOpen, Needs TriagePublicActions

Description

Related ObjectsSearch...

Event Timeline

Add failure rate triggered rollback to scap
Open, Needs TriagePublic
Actions

Related Objects
Search...