Page MenuHomePhabricator
Create Task
Maniphest T282922

Implement system-wide overall request rate / resource consumption limits in the orchestrator
Closed, ResolvedPublic
Actions

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Update function-schemata sub-module to HEAD (4583bb7)Jforrestermediawiki/extensions/WikiLambdamaster+1 -1
wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-17-184727Jforresteroperations/deployment-chartsmaster+4 -4
wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-16-205705Jforresteroperations/deployment-chartsmaster+4 -4
wikifunctions: Upgrade orchestrator from 2026-06-09-215338 to 2026-06-11-171152Jforresteroperations/deployment-chartsmaster+1 -1
wikifunctions: Upgrade orchestrator from 2026-05-12-210548 to 2026-05-19-145724Jforresteroperations/deployment-chartsmaster+1 -1
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
definitions: Add Z581/Reached outbound rate limit in orchestratorrepos/abstract-wiki/wikifunctions/function-schemata!366jforresterT282922-errormain
routes: Re-classify Z576/evaluator_wasi_limit as 422 at orchestrator edgerepos/abstract-wiki/wikifunctions/function-orchestrator!662jforresterT423383main
Customize query in GitLab

Related Objects
Search...

Event Timeline

Jdforrester-WMF created this task.May 15 2021, 12:44 AM
DVrandecic moved this task from To Triage to To triage: Un-phased work on the Abstract Wikipedia team board.May 26 2021, 4:43 PM
DVrandecic moved this task from To triage: Un-phased work to Phase θ – Throttling on the Abstract Wikipedia team board.Jul 14 2021, 4:47 PM
DVrandecic edited projects, added: Abstract Wikipedia team (Phase θ – Throttling); removed: Abstract Wikipedia team.
• OZhang-WMF moved this task from Incoming to Ready: G6. Stable and secure running (Performance) on the Abstract Wikipedia team (Phase θ – Throttling) board.Aug 2 2022, 12:39 AM
ori subscribed.Aug 4 2022, 3:22 PM

@Jdforrester-WMF what is the rationale for having a system-wide request-rate limit?

Jdforrester-WMF added a comment.Aug 4 2022, 3:25 PM
In T282922#8131543, @ori wrote:

@Jdforrester-WMF what is the rationale for having a system-wide request-rate limit?

Security and SRE were worried about out-of-control processes in Wikifunctions adding significant load to the production machines (especially if e.g. the calls from the orchestrator to Wikifunctions.org's MW API, or Wikidata.org's, got into a runaway loop) and could imperil the overall cluster stability.

cmassaro moved this task from Ready: G6. Stable and secure running (Performance) to Ready: G6. Stable & secure on the Abstract Wikipedia team (Phase θ – Throttling) board.Oct 5 2022, 3:52 PM
cmassaro added a project: function-orchestrator.
DVrandecic subscribed.Dec 7 2022, 11:56 PM

Check with Security if it is OK to simply use the Docker rate limits, instead of implementing our own solution. Contingent on that, moving this to nice to have.

James's estimated effort: 10+ days? Mostly blocked by tracking data about usage in the orchestrator.

DVrandecic moved this task from Phase θ – Throttling to To triage: Nice-to-haves on the Abstract Wikipedia team board.Dec 7 2022, 11:57 PM
DVrandecic edited projects, added: Abstract Wikipedia team; removed: Abstract Wikipedia team (Phase θ – Throttling).
cmassaro subscribed.Jan 18 2023, 2:54 PM

Are there any updates on this? What resources we want to track, strategies for avoiding the runaway cases mentioned by James?

Jdforrester-WMF added a comment.Jan 18 2023, 3:08 PM
In T282922#8535157, @cmassaro wrote:

Are there any updates on this? What resources we want to track, strategies for avoiding the runaway cases mentioned by James?

This is mostly moot until we have external resources we're requesting (i.e. Wikidata and Commons); I suppose we could build it as-is for requests to Wikifunctions's wiki API for now?

In my mind we'd just have all requests go through a request proxy, either just using the existing network service (which I think we have to use anyway?) or more smartly some code inside the orchestrator which keeps a running track of request rates and rejects/slows down requests over some threshold?

Jdforrester-WMF edited parent tasks, added: T338522: Implement agreed post-launch security and control measures in the Wikifunctions system; removed: T282913: Implement agreed security and control measures in the Wikifunctions system.Jun 8 2023, 9:31 PM
Jdforrester-WMF triaged this task as High priority.May 15 2024, 4:56 PM
Jdforrester-WMF moved this task from To triage: Nice-to-haves to Upcoming Epics on the Abstract Wikipedia team board.
Jdforrester-WMF moved this task from Upcoming Epics to Engineering Backlog on the Abstract Wikipedia team board.Nov 18 2024, 4:43 PM
Jdforrester-WMF added a project: Abstract Wikipedia Fix-It tasks.Sep 30 2025, 6:28 PM
CodeReviewBot added a project: Patch-For-Review.May 7 2026, 3:11 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/662

routes: Re-classify Z576/evaluator_wasi_limit as 422 at orchestrator edge

CodeReviewBot added a comment.May 18 2026, 3:22 PM

gengh merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/662

routes: Re-classify Z576/evaluator_wasi_limit as 422 at orchestrator edge

Maintenance_bot removed a project: Patch-For-Review.May 18 2026, 3:30 PM
gerritbot added a comment.May 19 2026, 3:11 PM

Change #1289373 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2026-05-12-210548 to 2026-05-19-145724

https://gerrit.wikimedia.org/r/1289373

gerritbot added a project: Patch-For-Review.May 19 2026, 3:11 PM
gerritbot added a comment.May 19 2026, 3:20 PM

Change #1289373 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2026-05-12-210548 to 2026-05-19-145724

https://gerrit.wikimedia.org/r/1289373

Maintenance_bot removed a project: Patch-For-Review.May 19 2026, 3:31 PM
CodeReviewBot added a project: Patch-For-Review.Jun 5 2026, 9:26 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/704

fetchObject: Cap (per-pod) concurrent requests to external APIs

Jdforrester-WMF changed the task status from Open to In Progress.Jun 5 2026, 9:26 PM
Jdforrester-WMF claimed this task.
Jdforrester-WMF edited projects, added: Abstract Wikipedia team (26Q4 (Apr–Jun)); removed: Abstract Wikipedia team.
Jdforrester-WMF added a project: Essential-Work.
Jdforrester-WMF moved this task from Incoming to In Code review on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.
Jdforrester-WMF mentioned this in T428676: Implement global process rate limit in orchestrator.Jun 10 2026, 5:27 PM
CodeReviewBot added a comment.Jun 10 2026, 10:06 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/366

definitions: Add Z581/Reached outbound rate limit in orchestrator

CodeReviewBot added a comment.Jun 10 2026, 11:02 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/366

definitions: Add Z581/Reached outbound rate limit in orchestrator

CodeReviewBot added a comment.Jun 10 2026, 11:04 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/712

Update function-schemata sub-module to HEAD (rECTXe8e00967df6a)

CodeReviewBot added a comment.Jun 10 2026, 11:44 PM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/712

Update function-schemata sub-module to HEAD (rECTXe8e00967df6a)

CodeReviewBot added a comment.Jun 11 2026, 1:15 AM

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/704

fetchObject: Cap (per-pod) concurrent requests to external APIs

Jdforrester-WMF moved this task from In Code review to Ready to deploy on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.Jun 11 2026, 1:19 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 11 2026, 1:31 AM
gerritbot added a comment.Jun 11 2026, 5:15 PM

Change #1300860 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2026-06-09-215338 to 2026-06-11-171152

https://gerrit.wikimedia.org/r/1300860

gerritbot added a project: Patch-For-Review.Jun 11 2026, 5:15 PM
gerritbot added a comment.Jun 11 2026, 5:22 PM

Change #1300860 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2026-06-09-215338 to 2026-06-11-171152

https://gerrit.wikimedia.org/r/1300860

Jdforrester-WMF moved this task from Ready to deploy to Needs Sign-off on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.Jun 11 2026, 5:28 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 11 2026, 5:31 PM
gerritbot added a comment.Jun 16 2026, 1:54 PM

Change #1302868 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (054c288)

https://gerrit.wikimedia.org/r/1302868

gerritbot added a project: Patch-For-Review.Jun 16 2026, 1:54 PM
CodeReviewBot added a comment.Jun 16 2026, 1:54 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/119

Update function-schemata sub-module to HEAD (054c288)

CodeReviewBot added a comment.Jun 16 2026, 1:55 PM

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/569

Update function-schemata sub-module to HEAD (054c288)

Jdforrester-WMF mentioned this in rMSFEfd77e2456a05: Update function-schemata sub-module to HEAD (054c288).Jun 16 2026, 1:56 PM
CodeReviewBot added a comment.Jun 16 2026, 1:58 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/119

Update function-schemata sub-module to HEAD (054c288)

CodeReviewBot added a comment.Jun 16 2026, 3:14 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/569

Update function-schemata sub-module to HEAD (054c288)

cmassaro mentioned this in rMSFE071c18079a82: Update function-schemata sub-module to HEAD (054c288).Jun 16 2026, 3:15 PM
gerritbot added a comment.Jun 17 2026, 1:49 PM

Change #1303433 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-16-205705

https://gerrit.wikimedia.org/r/1303433

gerritbot added a comment.Jun 17 2026, 2:04 PM

Change #1303433 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-16-205705

https://gerrit.wikimedia.org/r/1303433

gerritbot added a comment.Jun 17 2026, 4:09 PM

Change #1303482 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-17-154210

https://gerrit.wikimedia.org/r/1303482

gerritbot added a comment.Jun 17 2026, 9:04 PM

Change #1303482 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-06-09-174730 to 2026-06-17-184727

https://gerrit.wikimedia.org/r/1303482

gerritbot added a comment.Jun 18 2026, 1:14 PM

Change #1302868 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (4583bb7)

https://gerrit.wikimedia.org/r/1302868

Maintenance_bot removed a project: Patch-For-Review.Jun 18 2026, 1:31 PM
Jdforrester-WMF closed this task as Resolved.Jul 2 2026, 12:43 PM
Jdforrester-WMF merged a task: T428676: Implement global process rate limit in orchestrator.Jul 3 2026, 8:45 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits