Page MenuHomePhabricator

k8s Evaluator JS pods crash due to child processes error
Closed, ResolvedPublic

Description

We are seeing JS Evaluator pods crash due to child processes error and need to handle these:

...
{"@timestamp":"2026-03-28T01:42:55.870Z","ecs.version":"8.10.0","http":{"request":{"id":"e20ca2fe-0b3c-4011-ade3-176bbaaabded"}},"log.level":"info","message":"...finished calling execute in evaluator","service":{"name":"function-evaluator"},"[span.id](http://span.id/)":"6cf181ef44fcad3c","[trace.id](http://trace.id/)":"4a88441f972e635adcb6beb5f39ac9d8"}
{"@timestamp":"2026-03-28T01:42:55.870Z","ecs.version":"8.10.0","http":{"request":{"id":"b9c4f1b8-222d-4e1d-b7a4-91d923c2100c"}},"log.level":"info","message":"calling execute in evaluator...","service":{"name":"function-evaluator"},"[span.id](http://span.id/)":"622e85992e067852","[trace.id](http://trace.id/)":"4b77f03352b3be0fb347f241a15ef592"}
{"@timestamp":"2026-03-28T01:42:55.871Z","ecs.version":"8.10.0","http":{"request":{"id":"b9c4f1b8-222d-4e1d-b7a4-91d923c2100c"}},"log.level":"info","message":"calling execute in executor..., original time: 2026-03-28T01:42:55.870Z","service":{"name":"function-evaluator"},"[span.id](http://span.id/)":"9241736da6451efb","[trace.id](http://trace.id/)":"2216a4a79fc186dd9ee2958764ac24bf"}
{"@timestamp":"2026-03-28T01:42:55.871Z","ecs.version":"8.10.0","http":{"request":{"id":"b9c4f1b8-222d-4e1d-b7a4-91d923c2100c"}},"log.level":"info","message":"...finished calling execute in executor, original time: 2026-03-28T01:42:55.871Z","service":{"name":"function-evaluator"},"[span.id](http://span.id/)":"9241736da6451efb","[trace.id](http://trace.id/)":"2216a4a79fc186dd9ee2958764ac24bf"}
{"@timestamp":"2026-03-28T01:42:55.888Z","ecs.version":"8.10.0","log.level":"info","message":"executor process termination completed; child process PID: 220621","service":{"name":"function-evaluator"},"[span.id](http://span.id/)":"2372c608faf28177","[trace.id](http://trace.id/)":"ce0d0980831307b24023091845f290ac"}
{"@timestamp":"2026-03-28T01:42:55.903Z","ecs.version":"8.10.0","labels":{"requestId":"2ad39502-b400-4a03-975f-41ba5073ba85"},"log.level":"info","message":"...finished calling executor in evaluator","service":{"name":"function-evaluator"},"[span.id](http://span.id/)":"2372c608faf28177","[trace.id](http://trace.id/)":"ce0d0980831307b24023091845f290ac"}
node:internal/process/promises:394
triggerUncaughtException(err, true /* fromPromise */);
^

[Error: No matching pid found] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/proc/220634/stat'
}

Node.js v24.14.0

Note:
We typically see the Orchestrator pods crash/restart (due to 'OOMKilled'); not so much activity in the Evaluator. Checking today (3/30/26), we see all pods experiencing it. The Py Eval pods and Orchestrator are both due to OOM.

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
create and register logger for processesrepos/abstract-wiki/wikifunctions/function-evaluator!504ecarggrace/T421815/handle-process-errorsmain
Draft: handle process-level errorsrepos/abstract-wiki/wikifunctions/function-evaluator!503ecarggrace/T421815/handle-process-throwsmain
Customize query in GitLab

Event Timeline

Crash log from the other JS Eval pod:

{"@timestamp":"2026-03-30T09:38:21.888Z","ecs.version":"8.10.0","http":{"request":{"id":"fc21be51-e9fd-45e0-beaf-751eace31683"}},"log.level":"info","message":"calling execute in executor..., original time: 2026-03-30T09:38:21.887Z","service":{"name":"function-evaluator"},"span.id":"b4f42e4b358e19f0","trace.id":"abd520d5b6859adfa341ece801d1f3e7"}
{"@timestamp":"2026-03-30T09:38:21.888Z","ecs.version":"8.10.0","http":{"request":{"id":"fc21be51-e9fd-45e0-beaf-751eace31683"}},"log.level":"info","message":"...finished calling execute in executor, original time: 2026-03-30T09:38:21.888Z","service":{"name":"function-evaluator"},"span.id":"b4f42e4b358e19f0","trace.id":"abd520d5b6859adfa341ece801d1f3e7"}
{"@timestamp":"2026-03-30T09:38:21.903Z","ecs.version":"8.10.0","log.level":"info","message":"executor process termination completed; child process PID: 181326","service":{"name":"function-evaluator"},"span.id":"f95dd4fe44699b74","trace.id":"89d62603cba04a223c518d2014d7a958"}
{"@timestamp":"2026-03-30T09:38:21.909Z","ecs.version":"8.10.0","labels":{"requestId":"44826cf0-d8bf-4969-a6c6-ec46b033c2b0"},"log.level":"info","message":"...finished calling executor in evaluator","service":{"name":"function-evaluator"},"span.id":"f95dd4fe44699b74","trace.id":"89d62603cba04a223c518d2014d7a958"}
{"@timestamp":"2026-03-30T09:38:21.910Z","ecs.version":"8.10.0","labels":{"requestId":"fc21be51-e9fd-45e0-beaf-751eace31683"},"log.level":"info","message":"...finished calling executor in evaluator","service":{"name":"function-evaluator"},"span.id":"57c449e1f865fc84","trace.id":"19707637fd3082578e5f74b271960b0c"}
{"@timestamp":"2026-03-30T09:38:21.910Z","ecs.version":"8.10.0","http":{"request":{"id":"fc21be51-e9fd-45e0-beaf-751eace31683"}},"labels":{"errorResult":"{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z22\"},\"Z22K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z24\"},\"Z22K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z883\"},\"Z883K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z883K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"errors\"},\"K2\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z5\"},\"Z5K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z576\"},\"Z5K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z885\"},\"Z885K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z576\"}},\"Z576K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"1500\"}}}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationMemoryUsage\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"169.09 MiB\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationCpuUsage\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"360.348 ms\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationStartTime\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"2026-03-30T09:38:20.995Z\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationEndTime\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"2026-03-30T09:38:21.909Z\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationDuration\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"914 ms\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}},\"K1\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}},\"K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"evaluationHostname\"},\"K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"function-evaluator-javascript-evaluator-5dc7b87c76-t75nn\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z882\"},\"Z882K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6\"},\"Z882K2\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z1\"}}}}}}}}}}}}}","info":"{\"reentrant\":false,\"remainingTime\":-1774785637661,\"requestId\":\"fc21be51-e9fd-45e0-beaf-751eace31683\",\"codingLanguage\":\"javascript\",\"function\":{\"codeString\":\"function Z12681( Z12681K1 ) {\\n\\treturn Z12681K1.length;\\n}\",\"functionName\":\"Z12681\"},\"schemaVersion\":\"2.1.0\",\"serializer\":{\"code\":{\"codeString\":\"function Z13520( Z13520K1 ) {\\n\\treturn {\\n\\t\\t\\\"Z1K1\\\": {\\n\\t\\t\\t\\\"Z1K1\\\": \\\"Z9\\\",\\n\\t\\t\\t\\\"Z9K1\\\": \\\"Z13518\\\"\\n\\t\\t\\t\\n\\t\\t},\\n\\t\\t\\\"Z13518K1\\\": {\\n\\t\\t\\t\\\"Z1K1\\\": \\\"Z6\\\",\\n\\t\\t\\t\\\"Z6K1\\\": Z13520K1.toString()\\n\\t\\t}\\n\\t};\\n}\",\"functionName\":\"Z13520\"},\"isList\":false},\"functionArguments\":{\"Z12681K1\":{\"object\":{\"K1\":{\"Z6005K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6005K2\":{\"Z12K1\":{\"K1\":{\"Z11K1\":{\"Z60K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"en\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z60\"}},\"Z11K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"populous\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z11\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z11\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z11\"}}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z12\"}},\"Z6005K3\":{\"Z60K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"en\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z60\"}},\"Z6005K4\":{\"Z6091K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"Q34698\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6091\"}},\"Z6005K5\":{\"K1\":{\"Z6003K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P1343\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z6091K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"Q105359170\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6091\"}},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6008\"}}},\"Z6003K7\":{\"Z6020K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6021\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6020\"}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}},\"K2\":{\"K1\":{\"Z6003K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P5275\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"147933\"},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6008\"}}},\"Z6003K7\":{\"Z6020K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6021\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6020\"}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}},\"K2\":{\"K1\":{\"Z6003K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P12690\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"1279575\"},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6008\"}}},\"Z6003K7\":{\"Z6020K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6021\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6020\"}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}},\"K2\":{\"K1\":{\"Z6003K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P12718\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"0042940\"},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6008\"}}},\"Z6003K7\":{\"Z6020K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6021\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6020\"}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}},\"K2\":{\"K1\":{\"Z6003K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P12724\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"0041705\"},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6008\"}}},\"Z6003K7\":{\"Z6020K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6021\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6020\"}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}},\"K2\":{\"K1\":{\"Z6003K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P12726\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"0053967\"},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6008\"}}},\"Z6003K7\":{\"Z6020K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6021\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6020\"}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}},\"K2\":{\"K1\":{\"Z6003K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P12739\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"0649110\"},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6008\"}}},\"Z6003K7\":{\"Z6020K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6021\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6020\"}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}},\"K2\":{\"K1\":{\"Z6003K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P13258\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"11592131\"},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6008\"}}},\"Z6003K7\":{\"Z6020K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6021\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6020\"}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}},\"K2\":{\"K1\":{\"Z6003K1\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P5191\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z6095K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L1062260\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6095\"}},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6008\"}}},\"Z6003K7\":{\"Z6020K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6021\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6020\"}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6003\"}}},\"Z6005K6\":{\"K1\":{\"Z6006K1\":{\"Z6096K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393-S1\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6096\"}},\"Z6006K2\":{\"Z12K1\":{\"K1\":{\"Z11K1\":{\"Z60K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"en\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z60\"}},\"Z11K2\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"having a lot of people; having a high population\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z11\"}},\"K2\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z11\"}}},\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z11\"}}},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z12\"}},\"Z6006K3\":{\"K1\":{\"Z6003K1\":{\"Z6096K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"L339393-S1\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6096\"}},\"Z6003K2\":{\"Z6092K1\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"P12510\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6092\"}},\"Z6003K3\":{\"Z1K1\":\"Z6\",\"Z6K1\":\"7960974994\"},\"Z6003K4\":{\"Z6040K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6042\"},\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6040\"}},\"Z6003K5\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z6007\"}}},\"Z6003K6\":{\"Z1K1\":{\"Z1K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z7\"},\"Z7K1\":{\"Z1K1\":\"Z9\",\"Z9K1\":\"Z881\"},\"Z881K1\":{\"Z","method":"POST"},"log.level":"warn","message":"Returning HTTP status: 504 due to error in Evaluator","service":{"name":"function-evaluator"},"span.id":"57c449e1f865fc84","trace.id":"19707637fd3082578e5f74b271960b0c"}
node:internal/process/promises:394
    triggerUncaughtException(err, true /* fromPromise */);
    ^

[Error: ESRCH: no such process, read] {
  errno: -3,
  code: 'ESRCH',
  syscall: 'read'
}

Node.js v24.14.0
ecarg triaged this task as High priority.Tue, Mar 31, 5:10 AM
ecarg changed the task status from Open to In Progress.Wed, Apr 1, 11:31 PM

Change #1268966 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-04-01-092119 to 2026-04-06-224243

https://gerrit.wikimedia.org/r/1268966

Change #1268966 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2026-04-01-092119 to 2026-04-06-224243

https://gerrit.wikimedia.org/r/1268966

The process errors seem to be handled as they are not the reason for pods crashing anymore. We will keep an eye out.