Maniphest T406346

[26Q2] Orchestrator memory squashing
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Jdforrester-WMF
	Oct 3 2025, 3:05 PM

Tags

Referenced Files

None

Subscribers

Jdforrester-WMF

Description

Hypothesis:
If we can prevent 10x memory consumption spikes, the orchestrator will be better able to handle Wikidata objects, supporting the utility of Wikifunctions as a platform for Abstract Wikipedia.

Acceptance Criteria

At least 2–3 spike investigations into root causes orchestrator crashes and implement at least one intervention to improve memory consumption
Share findings about memory usage among the team to steer work
Thrice-weekly pair programming

Goals & Success Metrics

Node's GC memory usage is well understood and does not spike beyond service's memory limits
Ability to process 10 large Wikidata items in a single function call without crashing
3 further engineers are Certified Orchestrator Whisperers for the orchestrator and can participate in hypothesis work involving its performance

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		cmassaro	T406346 [26Q2] Orchestrator memory squashing
		Resolved	Spike	ecarg	T406926 Investigate potential memory leak
		Resolved	Spike	ecarg	T410813 create tool to trace garbage collection
		Resolved	Spike	ecarg	T410814 create tool to visualize memory usage
		Resolved	Spike	ecarg	T406928 Investigate cause of exceeding recursion limit
		Resolved		cmassaro	T406934 Establish Regular Pair Programming Times for Orchestrator Whisperer Apprenticeship
		Declined		None	T406935 Audit isValidated_ and isFullyResolved_ attributes of ZWrapper
		Resolved	Spike	ecarg	T407163 Investigate cause of exceeding rate limit
		Resolved	BUG REPORT	cmassaro	T406976 Z28602 often, but not always, fails with a Z570
		Open	BUG REPORT	None	T407827 Iterating over 456 Wikidata statements exceeds orchestrator time limit
		Open	BUG REPORT	None	T405849 "Reached time limit in orchestrator" error from calling Z28445/"Most recent year-specific sentence about item" on a country
		Resolved		DMartin-WMF	T408013 Arrange for caching of results from Wikidata haswbstatement searches
		Resolved		ecarg	T408751 Call setIsValidated after running validateAsType
		Resolved		cmassaro	T409379 create and add a test that counts calls to validation
		Resolved		Jdforrester-WMF	T409111 Use ORCHESTRATOR_CONFIG.maxSimultaneousExecutions in production
		Resolved		cmassaro	T407939 Provide a new config flag to control rate-limiting of simultaneity
		Open		None	T410065 Store Enriched Objects in Memcached

Event Timeline

Jdforrester-WMF created this task.Oct 3 2025, 3:05 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 3 2025, 3:05 PM

Jdforrester-WMF added a project: OKR-Work.Oct 7 2025, 2:58 PM

Dropping the brainstorm doc here!

ecarg changed the status of subtask T406926: Investigate potential memory leak from Open to In Progress.Oct 9 2025, 8:37 PM

cmassaro added a subtask: T406934: Establish Regular Pair Programming Times for Orchestrator Whisperer Apprenticeship.Oct 9 2025, 9:07 PM

cmassaro added a subtask: T406935: Audit isValidated_ and isFullyResolved_ attributes of ZWrapper.Oct 9 2025, 9:13 PM

ecarg changed the status of subtask T406928: Investigate cause of exceeding recursion limit from Open to In Progress.Oct 9 2025, 10:24 PM

ecarg changed the status of subtask T407163: Investigate cause of exceeding rate limit from Open to In Progress.Oct 13 2025, 11:25 PM

Jdforrester-WMF changed the status of subtask T406934: Establish Regular Pair Programming Times for Orchestrator Whisperer Apprenticeship from Open to In Progress.Oct 15 2025, 5:41 PM

Jdforrester-WMF added a subtask: T406976: Z28602 often, but not always, fails with a Z570.Oct 15 2025, 5:50 PM

cmassaro closed subtask T406934: Establish Regular Pair Programming Times for Orchestrator Whisperer Apprenticeship as Resolved.Oct 15 2025, 9:55 PM

Jdforrester-WMF triaged this task as High priority.Oct 16 2025, 2:11 PM

DMartin-WMF added a subtask: T407827: Iterating over 456 Wikidata statements exceeds orchestrator time limit.Oct 22 2025, 5:24 AM

DMartin-WMF added a subtask: T405849: "Reached time limit in orchestrator" error from calling Z28445/"Most recent year-specific sentence about item" on a country.Oct 22 2025, 5:26 AM

DMartin-WMF added a subtask: T408013: Arrange for caching of results from Wikidata haswbstatement searches.Oct 23 2025, 5:14 AM

ecarg added a subtask: T408751: Call setIsValidated after running validateAsType.Nov 2 2025, 5:48 PM

cmassaro added a subtask: T409111: Use ORCHESTRATOR_CONFIG.maxSimultaneousExecutions in production.Nov 3 2025, 8:18 PM

Jdforrester-WMF changed the status of subtask T408751: Call setIsValidated after running validateAsType from Open to In Progress.Nov 5 2025, 6:37 PM

Jdforrester-WMF removed a subtask: T407939: Provide a new config flag to control rate-limiting of simultaneity.Nov 5 2025, 8:01 PM

DMartin-WMF closed subtask T408013: Arrange for caching of results from Wikidata haswbstatement searches as Resolved.Nov 6 2025, 4:45 AM

DMartin-WMF mentioned this in T406682: Caching: Handle Wikidata objects bigger than 1 MB.Nov 10 2025, 11:39 PM

DSantamaria subscribed.Nov 11 2025, 5:06 PM

DSantamaria changed the status of subtask T406976: Z28602 often, but not always, fails with a Z570 from Open to In Progress.Nov 12 2025, 7:00 AM

ecarg closed subtask T408751: Call setIsValidated after running validateAsType as Resolved.Nov 14 2025, 10:02 PM

ecarg closed subtask T406926: Investigate potential memory leak as Resolved.Nov 18 2025, 5:52 AM

ecarg closed subtask T406928: Investigate cause of exceeding recursion limit as Resolved.Nov 18 2025, 9:20 PM

Jdforrester-WMF added a subtask: T410065: Store Enriched Objects in Memcached.Nov 19 2025, 6:33 PM

ecarg closed subtask T406976: Z28602 often, but not always, fails with a Z570 as Resolved.Nov 19 2025, 11:35 PM

ecarg reopened subtask T406926: Investigate potential memory leak as Open.

Jdforrester-WMF reopened subtask T406928: Investigate cause of exceeding recursion limit as In Progress.Nov 20 2025, 5:26 PM

Jdforrester-WMF changed the status of subtask T406926: Investigate potential memory leak from Open to In Progress.Nov 20 2025, 8:53 PM

Jdforrester-WMF closed subtask T406928: Investigate cause of exceeding recursion limit as Resolved.Dec 2 2025, 4:06 PM

Jdforrester-WMF changed the status of subtask T409111: Use ORCHESTRATOR_CONFIG.maxSimultaneousExecutions in production from Open to In Progress.Dec 3 2025, 3:19 PM

Jdforrester-WMF closed subtask T409111: Use ORCHESTRATOR_CONFIG.maxSimultaneousExecutions in production as Resolved.Dec 4 2025, 8:53 PM

ecarg closed subtask T406926: Investigate potential memory leak as Resolved.Dec 8 2025, 7:30 PM

ecarg closed subtask T407163: Investigate cause of exceeding rate limit as Resolved.Dec 9 2025, 3:46 AM

cmassaro closed subtask T406935: Audit isValidated_ and isFullyResolved_ attributes of ZWrapper as Declined.Dec 10 2025, 6:34 PM

Closed as unsupported.