⚓ T282920 Implement per-request timeout / resource consumption limits in the orchestrator

	Subject	Repo	Branch	Lines +/-
	Add example of orchestrator timeout.	mediawiki/extensions/WikiLambda	master	+197 -1
	Add twenty-second timeout to the orchestrator.	mediawiki/services/function-orchestrator	master	+46 -9

Status	Subtype	Assigned	Task
Open	PRODUCTION ERROR	None	T344206 Special:MathStatus exception error on Wikifunctions
Resolved		Jdforrester-WMF	T342865 Post-creation work for wikifunctionswiki
Resolved		BTullis	T289316 Prepare and check storage layer for Wikifunctions.org (new public content wiki)
Resolved		Jdforrester-WMF	T275945 Create Wikifunctions.org
Resolved		Jdforrester-WMF	T299176 Phase Theta – Root task
Resolved		cmassaro	T299596 Add security limits to the Wikifunctions system to maintain availability and load factors
Resolved		cmassaro	T282920 Implement per-request timeout / resource consumption limits in the orchestrator

Jdforrester-WMF created this task.May 15 2021, 12:43 AM

DVrandecic moved this task from To Triage to To triage: Un-phased work on the Abstract Wikipedia team board.May 26 2021, 4:43 PM

DVrandecic moved this task from To triage: Un-phased work to Phase θ – Throttling on the Abstract Wikipedia team board.Jul 14 2021, 4:48 PM

DVrandecic edited projects, added Abstract Wikipedia team (Phase θ – Throttling); removed Abstract Wikipedia team.

• OZhang-WMF moved this task from Incoming to Ready: G6. Stable and secure running (Performance) on the Abstract Wikipedia team (Phase θ – Throttling) board.Aug 5 2022, 3:28 AM

cmassaro moved this task from Ready: G6. Stable and secure running (Performance) to Ready: G6. Stable & secure on the Abstract Wikipedia team (Phase θ – Throttling) board.Oct 5 2022, 3:52 PM

This probably needs to be done.

James's estimated effort: 0.5 day (if whoever picks this up thinks that will take considerably longer, please check whether we agree on the task)

The timeout part should take 0.5 days, yes (1 at most), for just an unconfigurable 15-second timeout. I'm not clear on the resource consumption part. What are we measuring in terms of resource consumption? What's too much?

Change 881403 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[mediawiki/extensions/WikiLambda@master] Add example of orchestrator timeout.

https://gerrit.wikimedia.org/r/881403

gerritbot added a project: Patch-For-Review.Jan 18 2023, 1:21 PM

Change 881405 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[mediawiki/services/function-orchestrator@master] Add twenty-second timeout to the orchestrator.

https://gerrit.wikimedia.org/r/881405

Change 881405 merged by jenkins-bot:

[mediawiki/services/function-orchestrator@master] Add twenty-second timeout to the orchestrator.

https://gerrit.wikimedia.org/r/881405

Jdforrester-WMF mentioned this in rMSFOe0e4881d0fe2: Add twenty-second timeout to the orchestrator..Jan 18 2023, 3:11 PM

Change 881403 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Add example of orchestrator timeout.

https://gerrit.wikimedia.org/r/881403

In T282920#8534742, @cmassaro wrote:

The timeout part should take 0.5 days, yes (1 at most), for just an unconfigurable 15-second timeout. I'm not clear on the resource consumption part. What are we measuring in terms of resource consumption? What's too much?

When I wrote this, I was thinking a per-request memory limit or similar. Happy to declare this Resolved as-is though if you think this is good enough?

Maintenance_bot removed a project: Patch-For-Review.Jan 18 2023, 4:31 PM

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.20; 2023-01-23).Jan 18 2023, 5:00 PM

Hmm, I'm assuming we'll handle the memory limit bit in Kube. Let's keep this open for that (unless there's already another, more specific task to cover that).

cmassaro removed cmassaro as the assignee of this task.Jan 27 2023, 12:34 PM

cmassaro subscribed.

cmassaro added a subtask: T337589: Scopes Are Not Always Attached to ZObjects Correctly.May 26 2023, 8:12 PM

Jdforrester-WMF moved this task from Phase θ – Throttling to Phase λ – Launch on the Abstract Wikipedia team board.Jun 16 2023, 9:05 AM

Jdforrester-WMF edited projects, added Abstract Wikipedia team (Phase λ – Launch); removed Abstract Wikipedia team (Phase θ – Throttling).

Jdforrester-WMF moved this task from Incoming to Production tasks on the Abstract Wikipedia team (Phase λ – Launch) board.Jun 16 2023, 9:39 AM

Jdforrester-WMF added a subtask: T338272: Create Rate Limit in Evaluator/Orchestrator.Jun 21 2023, 3:27 PM

cmassaro added a parent task: T299596: Add security limits to the Wikifunctions system to maintain availability and load factors.Jun 26 2023, 3:48 PM

cmassaro removed a subtask: T337589: Scopes Are Not Always Attached to ZObjects Correctly.

cmassaro removed a parent task: T282913: Implement agreed security and control measures in the Wikifunctions system.

cmassaro removed a subtask: T338272: Create Rate Limit in Evaluator/Orchestrator.

cmassaro claimed this task.Jun 29 2023, 4:45 PM