wide-scale Python failure
Open, HighPublicBUG REPORT
Actions

Assigned To

None

Authored By

	99of9
	Oct 9 2025, 11:28 AM

Description

Steps to replicate the issue (include links if applicable):

go to https://www.wikifunctions.org/view/en/Z14287, and "try this implementation" with any string of digits, e.g. "321"

What happens?:

What should have happened instead?:
It should work. This is an incredibly simple python implementation. We have trouble with Python everywhere at the moment. This is just one example.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
wikifunctions: Upgrade evaluators from 2025-12-08-185405 to 2025-12-10-150641	operations/deployment-charts	master	+2 -2
wikifunctions: Add Python test call back in to test script	operations/deployment-charts	master	+4 -5
wikifunctions: Upgrade evaluators from 2025-10-28-150053 to 2025-11-05-063501	operations/deployment-charts	master	+2 -2

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	BUG REPORT	None	T406848 wide-scale Python failure
Resolved		cmassaro	T408823 Fail Early if Evaluator Can't Acquire a WASI Runner
Resolved		cmassaro	T409071 When evaluator fails due to WASI unavailability, consider queuing the runner back to the executor pool
Resolved		cmassaro	T408826 Make the evaluator fail early and respond with the appropriate Z500 and HTTP status code when WASI runners are unavailable
Resolved		cmassaro	T408825 Make the orchestrator react appropriately when the evaluator fails to acquire WASI runners
Resolved		cmassaro	T408824 Allocate new Z500 and HTTP error codes for failure to acquire a WASI runner
Resolved		Jdforrester-WMF	T408977 Set FUNCTION_EVALUATOR_WASI_ACQUIRE_TIMEOUT in production config
Open		None	T411481 Count all running subtasks of the evaluator when monitoring for process issues, not just those that contain "wasm"
Resolved		ecarg	T411744 add heartbeat metric to Evaluator
Open	Spike	ecarg	T408899 Change the function-evaluator readiness endpoint from _info to a new one that actually tells k8s if we're healthy for evaluation

Event Timeline

99of9 created this task.Oct 9 2025, 11:28 AM

99of9 mentioned this in T406784: When editing a test case, an outdated version of the code for amended implementations may be used for several hours.Oct 9 2025, 11:31 AM

cmassaro claimed this task.Oct 9 2025, 12:22 PM

I've deployed a new version of the evaluators, and this is now fixed.

In running the test command before deploying the above, it indeed failed:

{"Z1K1":"Z22","Z22K1":"Z24","Z22K2":{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z883","Z883K1":"Z6","Z883K2":"Z1"},"K1":[{"Z1K1":"Z7","Z7K1":"Z882","Z882K1":"Z6","Z882K2":"Z1"},{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z882","Z882K1":"Z6","Z882K2":"Z1"},"K1":"errors","K2":{"Z1K1":"Z5","Z5K1":"Z507","Z5K2":{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z885","Z885K1":"Z507"},"Z507K1":{"Z1K1":"Z99","Z99K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z8"},"Z8K1":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z17"}},"K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z17"},"Z17K1":{"Z1K1":"Z9","Z9K1":"Z6"},"Z17K2":{"Z1K1":"Z6","Z6K1":"Z400K1"},"Z17K3":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z12"},"Z12K1":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z11"}}}}},"K2":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z17"}},"K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z17"},"Z17K1":{"Z1K1":"Z9","Z9K1":"Z6"},"Z17K2":{"Z1K1":"Z6","Z6K1":"Z400K2"},"Z17K3":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z12"},"Z12K1":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z11"}}}}},"K2":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z17"}}}}},"Z8K2":{"Z1K1":"Z9","Z9K1":"Z1"},"Z8K3":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z20"}}},"Z8K4":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z14"}},"K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z14"},"Z14K1":{"Z1K1":"Z9","Z9K1":"Z400"},"Z14K3":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z16"},"Z16K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z61"},"Z61K1":{"Z1K1":"Z6","Z6K1":"python"}},"Z16K2":{"Z1K1":"Z6","Z6K1":"def Z400(Z400K1,Z400K2):\n\treturn str(int(Z400K1) + int(Z400K2))"}}},"K2":{"Z1K1":{"Z1K1":{"Z1K1":"Z9","Z9K1":"Z7"},"Z7K1":{"Z1K1":"Z9","Z9K1":"Z881"},"Z881K1":{"Z1K1":"Z9","Z9K1":"Z14"}}}},"Z8K5":{"Z1K1":"Z9","Z9K1":"Z400"}},"Z400K1":{"Z1K1":"Z6","Z6K1":"5"},"Z400K2":{"Z1K1":"Z6","Z6K1":"8"}}},"Z507K2":{"Z1K1":"Z5","Z5K1":"Z500","Z5K2":{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z885","Z885K1":"Z500"},"Z500K1":"Function evaluation failed with status 504: {\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z22\"},\"Z22K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z24\"},\"Z22K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z883\"},\"Z883K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z883K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"errors\"},\"K2\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z5\"},\"Z5K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z575\"},\"Z5K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z885\"},\"Z885K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z575\"}},\"Z575K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"9000 ms\"}}}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationMemoryUsage\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"123.33 MiB\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationCpuUsage\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"14.568 ms\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationStartTime\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"2025-10-09T13:30:46.487Z\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationEndTime\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"2025-10-09T13:30:55.487Z\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationDuration\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"9000 ms\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationHostname\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"function-evaluator-python-evaluator-f9fb846bc-tb8cw\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}}}}}}}}}}}}"}}}}},{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z882","Z882K1":"Z6","Z882K2":"Z1"},"K1":"orchestrationMemoryUsage","K2":"162.86 MiB"},{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z882","Z882K1":"Z6","Z882K2":"Z1"},"K1":"orchestrationCpuUsage","K2":"56.219 ms"},{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z882","Z882K1":"Z6","Z882K2":"Z1"},"K1":"orchestrationStartTime","K2":"2025-10-09T13:30:46.470Z"},{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z882","Z882K1":"Z6","Z882K2":"Z1"},"K1":"orchestrationEndTime","K2":"2025-10-09T13:30:55.494Z"},{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z882","Z882K1":"Z6","Z882K2":"Z1"},"K1":"orchestrationDuration","K2":"9024 ms"},{"Z1K1":{"Z1K1":"Z7","Z7K1":"Z882","Z882K1":"Z6","Z882K2":"Z1"},"K1":"orchestrationHostname","K2":"function-orchestrator-main-orchestrator-66bd767b7-r2hsm"}]}}

… or in short form, a Z507 (failed evaluation) with a Z575 (timeout), but no further details.

When running from the deployment server, I got "Orchestration generally failed," while I did indeed get a timeout when running from the UI. Very weird.

cmassaro removed cmassaro as the assignee of this task.Oct 9 2025, 2:39 PM

cmassaro subscribed.

Remaining work:

Document work-arounds
Determine the cause
Fix the issue

In T406848#11260208, @cmassaro wrote:

When running from the deployment server, I got "Orchestration generally failed," while I did indeed get a timeout when running from the UI. Very weird.

This is happening again. Do we need a new ticket?

Raised as T408638

In T406848#11320862, @GrounderUK wrote:

In T406848#11260208, @cmassaro wrote:

When running from the deployment server, I got "Orchestration generally failed," while I did indeed get a timeout when running from the UI. Very weird.

This is happening again. Do we need a new ticket?

No, we can work from this one.

We still need to do all the things James mentioned: determine the cause, fix the issue, etc. So this bug should remain open as a tracking bug until we've done all of that.

As for the current outage, I've scheduled a deployment for later in the day (00:00 UTC). That means we will hopefully be back up in three hours.

GrounderUK merged a task: T408638: Python DOWN: all Python evaluations on Wikifunctions.org fail with a timeout in orchestration.Oct 28 2025, 9:45 PM

Python appears to be up and running again.

cmassaro added a subtask: T408823: Fail Early if Evaluator Can't Acquire a WASI Runner.Oct 30 2025, 4:20 PM

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/436

Test concurrency in ExecutorPool and avoid stale locks.

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/436

Test concurrency in ExecutorPool and avoid stale locks.

DMartin-WMF mentioned this in rMSFE4adc20e65847: Test concurrency in ExecutorPool and avoid stale locks..Nov 5 2025, 3:45 AM

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2025, 4:30 AM

Change #1202148 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-10-28-150053 to 2025-11-05-063501

https://gerrit.wikimedia.org/r/1202148

gerritbot added a project: Patch-For-Review.Nov 5 2025, 2:25 PM

Change #1202148 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-10-28-150053 to 2025-11-05-063501

https://gerrit.wikimedia.org/r/1202148

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2025, 3:31 PM

DSantamaria changed the status of subtask T408823: Fail Early if Evaluator Can't Acquire a WASI Runner from Open to In Progress.Nov 12 2025, 6:59 AM

Change #1204603 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] [WIP] Add Python test call back in

https://gerrit.wikimedia.org/r/1204603

gerritbot added a project: Patch-For-Review.Nov 12 2025, 3:18 PM

Jdforrester-WMF added a subtask: T408977: Set FUNCTION_EVALUATOR_WASI_ACQUIRE_TIMEOUT in production config.Nov 12 2025, 3:23 PM

Change #1204603 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Add Python test call back in to test script

https://gerrit.wikimedia.org/r/1204603

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2025, 4:30 PM

cmassaro closed subtask T408823: Fail Early if Evaluator Can't Acquire a WASI Runner as Resolved.Nov 23 2025, 1:31 AM

Recurrence reported on Telegram at 09:12 UTC on 2025-11-30.

Although an emergency fix has been deployed with some success (thanks), we are still getting random, intermittent timeouts like this one.

This succeeded immediately after this failed.

Please also see integers have the same sign, python (Z17272), where the first and third tests in the list failed and other tests passed, at the more or less the same time.

Jdforrester-WMF mentioned this in T411481: Count all running subtasks of the evaluator when monitoring for process issues, not just those that contain "wasm".Dec 2 2025, 3:10 PM

cmassaro added a subtask: T411481: Count all running subtasks of the evaluator when monitoring for process issues, not just those that contain "wasm".Dec 2 2025, 3:38 PM

GrounderUK mentioned this in T411597: Concatenation of several literal strings times out.Dec 3 2025, 11:04 AM

CDanis subscribed.Dec 3 2025, 2:09 PM

Jdforrester-WMF changed the status of subtask T408977: Set FUNCTION_EVALUATOR_WASI_ACQUIRE_TIMEOUT in production config from Open to In Progress.Dec 3 2025, 3:23 PM

Jdforrester-WMF merged a task: T411597: Concatenation of several literal strings times out.Dec 3 2025, 6:48 PM

Jdforrester-WMF added a subscriber: DVrandecic.

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/450

Executor: Rename immaHeadOut to killChildrenProcessIfStillAround; add logs; group code

Jdforrester-WMF mentioned this in rMSFEd654b2fcded7: Executor: Rename immaHeadOut to killChildrenProcessIfStillAround; add logs….Dec 4 2025, 5:09 PM

Jdforrester-WMF mentioned this in rMSFEdfd26cf4a3fc: Executor: Rename immaHeadOut to killChildProcessesIfStillAround; add logs….

So9q subscribed.Dec 4 2025, 6:31 PM