Maniphest T407588

Create a fail-fast for rate-limiting
Closed, ResolvedPublic
Actions

Assigned To

Jdforrester-WMF

Authored By

	ecarg
	Oct 16 2025, 11:47 PM

Project Tags

Referenced Files

None

Subscribers

Description

While adding rate-limiting to the Rust-based evaluator (MR), we discovered that Rust libraries do not automatically fail with 429's when the rate limit is reached and queues the request instead. We want to fail with the standard error code and in order to do that will require some custom middleware and/or incorporating other libraries to fail after a certain threshold.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	wikifunctions: Upgrade evaluators from 2026-03-10-214300 to 2026-03-16-124858	Jforrester	operations/deployment-charts	master	+2 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		DSantamaria	T421849 [Hypothesis] WE2.3.16 Productionize Rust Evaluator and Deprecate Node Evaluator
		Resolved		Jdforrester-WMF	T382433 Get the Rust prototype of the function-evaluator's executors into a state where it could be deployed
		Resolved		Jdforrester-WMF	T343720 If we create a working evaluator service in Rust, we can measure its performance and stability characteristics and plan to productionize it in a subsequent quarter
		Resolved		cmassaro	T399344 Rust-based evaluator is at feature parity with the current Node-based one
		Resolved		cmassaro	T399322 Rust-based evaluator service should maintain a pool of ready executors
		Resolved		cmassaro	T399323 Rust-based evaluator can add all currently-supported metadata
		Resolved		ecarg	T399345 Rust-based evaluator implements a timeout
		Resolved		ecarg	T399346 Rust-based evaluator implements rate limiting
		Resolved		cmassaro	T399321 Rust-based evaluator service can run a function call in the appropriate executor and return a result
		Resolved		cmassaro	T400820 Use dedicated abstract-wikipeda-rust image in test-rust (and, eventually, other Rust images)
		Resolved		Jdforrester-WMF	T417870 Specify Rust and Node versions of the evaluators in prod and staging configs
		Resolved		Jdforrester-WMF	T402957 [26Q3] Rust evaluator in production
		Resolved		Jdforrester-WMF	T407588 Create a fail-fast for rate-limiting
		Resolved		ecarg	T407590 Check that timeout is working properly

Event Timeline

ecarg created this task.Oct 16 2025, 11:47 PM

ecarg triaged this task as Medium priority.

ecarg added a parent task: T343720: If we create a working evaluator service in Rust, we can measure its performance and stability characteristics and plan to productionize it in a subsequent quarter.

ecarg added a parent task: T402957: [26Q3] Rust evaluator in production.

ecarg added a project: Essential-Work.

ecarg added a parent task: T399346: Rust-based evaluator implements rate limiting.Oct 16 2025, 11:56 PM

cmassaro edited projects, added: Abstract Wikipedia team (26Q2 (Oct–Dec)); removed: Abstract Wikipedia team.Oct 22 2025, 5:37 PM

cmassaro moved this task from Incoming to Ready on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.Oct 22 2025, 5:58 PM

cmassaro edited projects, added: Abstract Wikipedia team (26Q3 (Jan–Mar)); removed: Abstract Wikipedia team (26Q2 (Oct–Dec)).Dec 17 2025, 2:30 PM

Jdforrester-WMF moved this task from Incoming to Ready on the Abstract Wikipedia team (26Q3 (Jan–Mar)) board.Jan 7 2026, 6:52 PM

Jdforrester-WMF changed the task status from Open to In Progress.Feb 6 2026, 3:22 PM

Jdforrester-WMF claimed this task.

Jdforrester-WMF moved this task from Ready to In Code review on the Abstract Wikipedia team (26Q3 (Jan–Mar)) board.

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/469

Replace tower RateLimitLayer with custom middleware returning HTTP 429.

Jdforrester-WMF mentioned this in rMSFEe4f171de2a29: Replace tower RateLimitLayer with custom middleware returning HTTP 429..Feb 6 2026, 3:25 PM

Jdforrester-WMF mentioned this in rMSFEa5e2fdab6bd9: Replace tower RateLimitLayer with custom middleware returning HTTP 429..Feb 10 2026, 5:35 PM

Jdforrester-WMF mentioned this in rMSFEf97d04d766f8: Replace tower RateLimitLayer with custom middleware returning HTTP 429..Feb 17 2026, 6:58 PM

cmassaro mentioned this in rMSFE8342a85d43fd: Replace tower RateLimitLayer with custom middleware returning HTTP 429..Mar 11 2026, 5:37 PM

cmassaro mentioned this in rMSFE72f59725f2b4: Replace tower RateLimitLayer with custom middleware returning HTTP 429..Mar 13 2026, 3:06 PM

cmassaro mentioned this in rMSFEea9b8f1bf513: Replace tower RateLimitLayer with custom middleware returning HTTP 429..Mar 13 2026, 5:26 PM

Jdforrester-WMF moved this task from In Code review to Ready to deploy on the Abstract Wikipedia team (26Q3 (Jan–Mar)) board.Mar 13 2026, 5:39 PM

Change #1254928 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-03-10-214300 to 2026-03-16-124858

https://gerrit.wikimedia.org/r/1254928

Change #1254928 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-03-10-214300 to 2026-03-16-124858

https://gerrit.wikimedia.org/r/1254928

Jdforrester-WMF closed this task as Resolved.Mar 18 2026, 2:17 PM

Jdforrester-WMF moved this task from Ready to deploy to Needs Sign-off on the Abstract Wikipedia team (26Q3 (Jan–Mar)) board.