Maniphest T413847

Implement Evaluator functionality for Evaluator-Orchestrator Callbacks
Open, In Progress, MediumPublic
Actions

Assigned To

Authored By

	DMartin-WMF
	Jan 6 2026, 3:24 AM

Project Tags

Referenced Files

None

Subscribers

Description

Description

Create code in evaluator that

~~Provides a Web socket server when a code object is marked as containing callbacks~~
- Update March 2026: web sockets have been replaced by a new HTTP endpoint provided by the orchestrator.
Receives callback requests from executors
Checks that they are valid requests
If so, transforms those requests into function-call ZObjects
Passes the ZObjects to the Orchestrator
Receives the function-call result from the Orchestrator, and returns it to the executor
If an evaluated implementation attempts to call a disallowed function, returns a suitable error.

Desired behavior/Acceptance criteria (returned value, expected error, performance expectations, etc.)

Completion checklist

Before closing this task, review one by one the checklist available here: https://www.mediawiki.org/wiki/Abstract_Wikipedia_team/Definition_of_Done#Back-end_Task/Bug_completion_checklist

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Draft: Support (limited) use of Evaluator-Orchestrator callbacks	repos/abstract-wiki/wikifunctions/function-evaluator!484	dmartin	T413847-2	main
	Draft: Support (initial, limited) use of Evaluator-Orchestrator callbacks	repos/abstract-wiki/wikifunctions/function-evaluator!475	dmartin	T413847	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T359566 Support re-entrancy
		Open		DMartin-WMF	T421848 [Hypothesis] WE2.3.15 Complete implementation and deployment of Evaluator-Orchestrator callbacks
		In Progress		DMartin-WMF	T413847 Implement Evaluator functionality for Evaluator-Orchestrator Callbacks

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

DMartin-WMF mentioned this in rMSFE24391e6a240d: Add callback tests with disallowed function ZID.Thu, Jun 4, 4:37 AM

DMartin-WMF mentioned this in rMSFE45c33965cce7: Remove inessential backslashes escaping quotes from a string.

DMartin-WMF mentioned this in rMSFEeacc297d3134: Revise OrchestratorStub to index callback results on the function call.

DMartin-WMF mentioned this in rMSFE28736954a281: Add Python test with bad argument in callback.

DMartin-WMF mentioned this in rMSFEb472353c5596: Remove unneeded expectedOutput sections from 4 test files.

DMartin-WMF mentioned this in rMSFE61e3d781f76c: Add 2 new Python callback tests & a new test directory.

DMartin-WMF mentioned this in rMSFE44671ecf2f89: Reduce length of some comment lines..

DMartin-WMF mentioned this in rMSFE4fe9eb7f1ce9: Add eslint-disable max-len for a temporary comment block.

DMartin-WMF mentioned this in rMSFE1a15ba378bde: Reorganize Wikidata-related callbacks test files.

DMartin-WMF mentioned this in rMSFEa68de6fcac45: Add 2 JavaScript callback tests.

DMartin-WMF mentioned this in rMSFE085469ed192d: Put the callback fetch inside a try block.

DMartin-WMF mentioned this in rMSFEd7704851f373: Fix JS Wikifunctions.Call exception handling related to callback results.

DMartin-WMF mentioned this in rMSFEbe38c15f9a0f: Add a test checking that a malformed callback result returns Z507.

DMartin-WMF mentioned this in rMSFE3337abed0f0b: Add timeout to orchestrator callback fetch.

DMartin-WMF mentioned this in rMSFEcddd5e928e2e: Add size limit to executor output sections in Executor.js.

DMartin-WMF mentioned this in rMSFE0979bdd5777d: Fix TypeError in processNonReady caused by calling reduce on QueueByLogging.

DMartin-WMF mentioned this in rMSFE01fa4f72915d: Add a test for triggering the "second reentrant call while fetch is pending"….

DMartin-WMF mentioned this in rMSFE82d387117797: Change exchange_format_2_1_0/ to ..._2_1_1/ for callback tests.

DMartin-WMF mentioned this in rMSFE0ee5fded5c41: Check functionCallRequest.allowsCallbacks before running a call back.

DMartin-WMF mentioned this in rMSFE5373a2c4c880: Change "reentrant" to "allowsCallbacks" in new JSON test files.

DMartin-WMF mentioned this in rMSFE0c5a2b9b898c: Update a test to include allowsCallbacks_ = true.

DMartin-WMF mentioned this in rMSFE77516a409c45: Update require paths for new location of function-schemata.

DMartin-WMF mentioned this in rMSFE4953a483f025: Support (initial, limited) use of Evaluator-Orchestrator callbacks.Fri, Jun 5, 1:12 PM

DMartin-WMF mentioned this in rMSFE8e69761d15bc: Change call to Error to self.Error.

DMartin-WMF mentioned this in rMSFEfe3907e47c0f: Update Wikifunctions.Call test to use an allowed function.

DMartin-WMF mentioned this in rMSFEc7fb325ff909: Eliminate extra call to getOrchestratorCallbackUri.

DMartin-WMF mentioned this in rMSFE7bc0a2abe604: Add ./lib folder to test-unit config.

DMartin-WMF mentioned this in rMSFE067e248af412: Remove remaining references to web sockets in test files.

DMartin-WMF mentioned this in rMSFE8180a8a381a7: Add testing with a mock orchestrator callback endpoint.

DMartin-WMF mentioned this in rMSFE67c60c9f8041: Add ParseState.AWAITING_REENTRANT_CALL to prevent concurrent callbacks.

DMartin-WMF mentioned this in rMSFEb29af64817eb: Add new JSON test file.

DMartin-WMF mentioned this in rMSFEceb4c06ea078: Update wikifunctions.js for callbacks (add .Call function).

DMartin-WMF mentioned this in rMSFE10391c8f273b: Adopt a simpler approach to preventing multiple concurrent callbacks.

DMartin-WMF mentioned this in rMSFE9a6a4b9c994b: Revert error string to what it was before.

DMartin-WMF mentioned this in rMSFEca8d8096f39c: Cleanup some TODO Comments.

DMartin-WMF mentioned this in rMSFE8327ebccb9a9: Add a JavaScript callbacks test.

DMartin-WMF mentioned this in rMSFE25257539c59e: Add two callback test cases.

DMartin-WMF mentioned this in rMSFE6e98b4214867: Improve the formatting of the disallowed_callback error.

DMartin-WMF mentioned this in rMSFEdaa26559a528: Remove unneeded import from wikifunctions.py.

DMartin-WMF mentioned this in rMSFE98aece9ded5c: Add callback tests with disallowed function ZID.

DMartin-WMF mentioned this in rMSFEdc0f196d3df6: Remove inessential backslashes escaping quotes from a string.

DMartin-WMF mentioned this in rMSFE4faaaf64578d: Revise OrchestratorStub to index callback results on the function call.

DMartin-WMF mentioned this in rMSFE9cbdb87e0bd8: Add Python test with bad argument in callback.

DMartin-WMF mentioned this in rMSFE1b1bf6557d34: Remove unneeded expectedOutput sections from 4 test files.

DMartin-WMF mentioned this in rMSFEb8b5cf8cd8b4: Reduce length of some comment lines..

DMartin-WMF mentioned this in rMSFEa7b54a172247: Add 2 new Python callback tests & a new test directory.

DMartin-WMF mentioned this in rMSFE50ab4d5eea0b: Add eslint-disable max-len for a temporary comment block.

DMartin-WMF mentioned this in rMSFE2da85954ae92: Reorganize Wikidata-related callbacks test files.

DMartin-WMF mentioned this in rMSFEfa73b699defd: Add 2 JavaScript callback tests.

DMartin-WMF mentioned this in rMSFE2d0dfaba4a95: Put the callback fetch inside a try block.

DMartin-WMF mentioned this in rMSFE8f5a1ee417a1: Fix JS Wikifunctions.Call exception handling related to callback results.

DMartin-WMF mentioned this in rMSFE20152b0c5f03: Add a test checking that a malformed callback result returns Z507.

DMartin-WMF mentioned this in rMSFE02e629b9ff73: Add timeout to orchestrator callback fetch.

DMartin-WMF mentioned this in rMSFE2dfa8e539f13: Add size limit to executor output sections in Executor.js.

DMartin-WMF mentioned this in rMSFE924951c362f4: Fix TypeError in processNonReady caused by calling reduce on QueueByLogging.

DMartin-WMF mentioned this in rMSFE11f76f9a7475: Add a test for triggering the "second reentrant call while fetch is pending"….

DMartin-WMF mentioned this in rMSFE1b8950626a39: Change exchange_format_2_1_0/ to ..._2_1_1/ for callback tests.

DMartin-WMF mentioned this in rMSFEbb65a4815487: Check functionCallRequest.allowsCallbacks before running a call back.

DMartin-WMF mentioned this in rMSFEff10506a636e: Change "reentrant" to "allowsCallbacks" in new JSON test files.

DMartin-WMF mentioned this in rMSFEe2004bb2274a: Update a test to include allowsCallbacks_ = true.

DMartin-WMF mentioned this in rMSFEd67243976f34: Update require paths for new location of function-schemata.

DMartin-WMF mentioned this in rMSFE884220fbef22: Change call to Error to self.Error.Mon, Jun 8, 3:20 PM

DMartin-WMF mentioned this in rMSFE3be6a2878ff7: Support (initial, limited) use of Evaluator-Orchestrator callbacks.

DMartin-WMF mentioned this in rMSFEd4eb91485360: Update Wikifunctions.Call test to use an allowed function.

DMartin-WMF mentioned this in rMSFE9800de4a1e72: Eliminate extra call to getOrchestratorCallbackUri.

DMartin-WMF mentioned this in rMSFEf443e4d23836: Add ./lib folder to test-unit config.

DMartin-WMF mentioned this in rMSFEce2039c675b3: Remove remaining references to web sockets in test files.

DMartin-WMF mentioned this in rMSFE00d29a69d6a8: Add testing with a mock orchestrator callback endpoint.

DMartin-WMF mentioned this in rMSFE5ffcfc2c0410: Add ParseState.AWAITING_REENTRANT_CALL to prevent concurrent callbacks.

DMartin-WMF mentioned this in rMSFE393981e8e502: Add new JSON test file.

DMartin-WMF mentioned this in rMSFE3e0a92ed2c0f: Update wikifunctions.js for callbacks (add .Call function).

DMartin-WMF mentioned this in rMSFE37d115015f1d: Adopt a simpler approach to preventing multiple concurrent callbacks.

DMartin-WMF mentioned this in rMSFE371071058b33: Revert error string to what it was before.

DMartin-WMF mentioned this in rMSFEf5d8f589690b: Cleanup some TODO Comments.

DMartin-WMF mentioned this in rMSFE3758b626358a: Add a JavaScript callbacks test.

DMartin-WMF mentioned this in rMSFE5b1787f2d018: Add two callback test cases.

DMartin-WMF mentioned this in rMSFEb4d666de6721: Improve the formatting of the disallowed_callback error.

DMartin-WMF mentioned this in rMSFEa0b1947b7a55: Remove unneeded import from wikifunctions.py.

DMartin-WMF mentioned this in rMSFE8f897fa829fa: Add callback tests with disallowed function ZID.

DMartin-WMF mentioned this in rMSFE35c6d0031dd7: Remove inessential backslashes escaping quotes from a string.

DMartin-WMF mentioned this in rMSFE1d0237c0e4df: Revise OrchestratorStub to index callback results on the function call.

DMartin-WMF mentioned this in rMSFE1792add42ba3: Add Python test with bad argument in callback.

DMartin-WMF mentioned this in rMSFE5f024fdc077f: Remove unneeded expectedOutput sections from 4 test files.

DMartin-WMF mentioned this in rMSFE4a125ccdfef2: Reduce length of some comment lines..

DMartin-WMF mentioned this in rMSFE3fc8cd053f4d: Add 2 new Python callback tests & a new test directory.

DMartin-WMF mentioned this in rMSFEe38591a5d830: Add eslint-disable max-len for a temporary comment block.

DMartin-WMF mentioned this in rMSFE1e97811c857f: Reorganize Wikidata-related callbacks test files.

DMartin-WMF mentioned this in rMSFE34844fd1932c: Put the callback fetch inside a try block.

DMartin-WMF mentioned this in rMSFE3f3879da79df: Add 2 JavaScript callback tests.

DMartin-WMF mentioned this in rMSFE3aeef27b2a37: Fix JS Wikifunctions.Call exception handling related to callback results.

DMartin-WMF mentioned this in rMSFE3aa2539bda88: Add a test checking that a malformed callback result returns Z507.

DMartin-WMF mentioned this in rMSFEc9d5e267ebae: Add timeout to orchestrator callback fetch.

DMartin-WMF mentioned this in rMSFE6c2fddaf97d4: Add size limit to executor output sections in Executor.js.

DMartin-WMF mentioned this in rMSFEbb8e4617710e: Fix TypeError in processNonReady caused by calling reduce on QueueByLogging.

DMartin-WMF mentioned this in rMSFE12e6dd2be418: Add a test for triggering the "second reentrant call while fetch is pending"….

DMartin-WMF mentioned this in rMSFE9665c3d37cab: Change exchange_format_2_1_0/ to ..._2_1_1/ for callback tests.

DMartin-WMF mentioned this in rMSFEcff99e9d1245: Check functionCallRequest.allowsCallbacks before running a call back.

DMartin-WMF mentioned this in rMSFEd869a796f130: Change "reentrant" to "allowsCallbacks" in new JSON test files.

DMartin-WMF mentioned this in rMSFE7fc8f808a930: Update a test to include allowsCallbacks_ = true.

DMartin-WMF mentioned this in rMSFEd8379327f85f: Update require paths for new location of function-schemata.