Maniphest T317405

Add failure rate triggered rollback to scap
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Clement_Goubert
	Sep 9 2022, 10:45 AM

Project Tags

Referenced Files

None

Subscribers

Clement_Goubert

Description

scap currently only stops if a canary hosts fail deployment.

I suggest that, further, when a certain percentage of failure to restart or repool is being reported by scap for a cluster after canaries pass, it should:

either stop or rollback
instruct on contact information for checking failure cause

Percentage of failure tbd.

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Add production error rate checks	repos/releng/scap!1159	dancy	master-I2773f10b349f3f41de0cb2724ab070bb099556fd	master
	logstash_checker.py: Make "scap analyze-logstash production" work better	repos/releng/scap!1157	dancy	master-I7d4afb1fbe9d37b14feb97f4723bc3997c0aba5d	master
	logstash_checker.py: Support arbitrary stage error rate checking	repos/releng/scap!1154	dancy	master-I35720c20c3c6f818bd5824924bcfdd1e717d809d	master

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		jijiki	T317340 Incident: 2022-09-08 codfw appservers degradation
		Open		None	T317405 Add failure rate triggered rollback to scap

Event Timeline

Clement_Goubert created this task.Sep 9 2022, 10:45 AM

Clement_Goubert updated the task description. (Show Details)Sep 9 2022, 3:44 PM

jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops-deprecated board.Sep 28 2022, 2:17 PM

Volans added a project: SRE-Sprint-Week-Sustainability-March2023.Mar 20 2023, 3:44 PM

Volans moved this task from Backlog to E_TOO_BIG_MAYBE_OKR? on the SRE-Sprint-Week-Sustainability-March2023 board.

Re-tagging to the team responsible for scap.

dancy mentioned this in T239376: Run Swagger checks in Scap before exposing to prod MW traffic .Feb 15 2024, 9:27 PM

dancy updated https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1154

logstash_checker.py: Support arbitrary stage error rate checking

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1154

logstash_checker.py: Support arbitrary stage error rate checking

dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1157

logstash_checker.py: Make "scap analyze-logstash production" work better

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1157

logstash_checker.py: Make "scap analyze-logstash production" work better

dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1159

Add production error rate checks

Scott_French subscribed.May 1 2026, 4:19 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1159

Add production error rate checks

Mentioned in SAL (#wikimedia-operations) [2026-05-01T19:37:28Z] <dancy@deploy1003> Started scap sync-world: testing T317405

Mentioned in SAL (#wikimedia-operations) [2026-05-01T19:40:51Z] <dancy@deploy1003> Finished scap sync-world: testing T317405 (duration: 03m 23s)

bd808 mentioned this in T428971: Allow configuration of canary and production checks based on deployment target.Jun 11 2026, 10:29 PM