• maybe we can start by adding a null check(s): e.g. url.path = info.request?.params?.[ 0 ];

I tried these in our Sandbox (Prod):

• OG: [https://www.wikifunctions.org/w/api.php?action=wikilambda_perform_test&format=json&wikilambda_perform_test_zfunction=Z12864&wikilambda_perform_test_zimplementations=Z12865|Z13394|Z13890&wikilambda_perform_test_ztesters=Z12866|Z13395|Z18127&uselang=en](https://www.wikifunctions.org/w/api.php?action=wikilambda_perform_test&format=json&wikilambda_perform_test_zfunction=Z12864&wikilambda_perform_test_zimplementations=Z12865%7CZ13394%7CZ13890&wikilambda_perform_test_ztesters=Z12866%7CZ13395%7CZ18127&uselang=en)
  • identical, using SB: https://www.wikifunctions.org/wiki/Special:ApiSandbox#action=wikilambda_perform_test&format=json&wikilambda_perform_test_zfunction=Z12864&wikilambda_perform_test_zimplementations=Z12865 |Z13394|Z13890&wikilambda_perform_test_ztesters=Z12866 |Z13395|Z18127&formatversion=2
    • we get a visible response body here too.
    • 500 from Orch: https://logstash.wikimedia.org/app/dashboards#/doc/0fade920-6712-11eb-8327-370b46f9e7a5/ecs-k8s-1-1.11.0-7-2025.18?id=qVdojpYBaJATmKuITY79
• another one w/ one less 'zimplementation':[https://www.wikifunctions.org/wiki/Special:ApiSandbox#action=wikilambda_perform_test&format=json&wikilambda_perform_test_zfunction=Z12864&wikilambda_perform_test_zimplementations=Z12865 |Z13394&wikilambda_perform_test_ztesters=Z12866 |Z13395|Z18127&formatversion=2](https://www.wikifunctions.org/wiki/Special:ApiSandbox#action=wikilambda_perform_test&format=json&wikilambda_perform_test_zfunction=Z12864&wikilambda_perform_test_zimplementations=Z12865%20%7CZ13394&wikilambda_perform_test_ztesters=Z12866%20%7CZ13395%7CZ18127&formatversion=2)
  • 500 error raised as a result, from Orch: https://logstash.wikimedia.org/app/dashboards#/doc/0fade920-6712-11eb-8327-370b46f9e7a5/ecs-k8s-1-1.11.0-7-2025.18?id=8Z9ijpYBF1zH-YDa2NOJ

Notes:

• I think this shows that those mysterious (https://phabricator.wikimedia.org/T388203) 500: [object Object] errors we've been seeing the past few weeks may also be related (and hopefully resolved alongside this)
• However, I don't see the 504 Gateway Timeout from running these requests in SB
• At best, I think the solution is twofold:
  1. Handle potential null case(s) and/or in service-utils
  2. Find where we could be trying to log huge JSON objects and stop doing that.

ecarg updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/341

Draft: add safe navigation + null coalesce

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/359

Draft: null handling

I think this is related to the huge request issue that @gengh is looking at. I am gonna try to work on point 2. (unless you were already doing it, @ecarg )?

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/360

Do not include expanded ZObjects in error output.

On the service-utils side, that property should've been filled in by express:

When you use a regular expression for the route definition, capture groups are provided in the array using req.params[n], where n is the nth capture group. This rule is applied to unnamed wild card matches with string routes such as /file/*

Regex routes are used by service-template-node. There's probably some route that isn't being captured here that needs some sensible default now that it isn't technically on service-template-node.

So, we're running into an issue here with how we retrieve type converters.

In a single step, we turn a Z7 into a binary-formatted evaluator request. This means that we don't have any options about the object that actually ends up in that request--we can either resolve everything (in order to have access to, e.g., the declared type converters), or we can not to do that and break things.

We should instead resolve everything, extract the required information, then create a stripped-down version of the arguments (crucially, with the types not fully resolved). We can then assemble the evaluator request and convert it to binary form in a separate step.

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/service-utils/-/merge_requests/13

Optional chain logging request properties

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/360

Do not include expanded ZObjects in error output.

As a final step, I am working to prevent us from shipping huge type definitions from the orchestrator to the evaluator.

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/219

Add a new set of functions to allow ZObject argument types to be spliced.

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/363

Draft: Minimize types as much as possible before sending to the evaluator.

Change #1143075 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-04-23-134615 to 2025-05-06-142345

https://gerrit.wikimedia.org/r/1143075

Change #1143075 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-04-23-134615 to 2025-05-06-142345

https://gerrit.wikimedia.org/r/1143075

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/219

Add a new set of functions to allow ZObject argument types to be spliced.

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/366

Update function-schemata sub-module to HEAD (cf50aca)

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/341

initApp: Handle null config

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/359

initApp: Handle null config

@cmassaro as this is causing

Mishandled responses from our logger is causing 500 outages in our Evaluator (both JS and Py).

Shouldn't it be either a "bug" or a "Production error"?

I suppose so! I wasn't the original filer of this task, but I would have no complaints about having it re-classified.

Current status: the immediate bug will be resolved once https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/366 and https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/363 are merged. The long-term solution is described in the document I linked in Slack.

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/366

Update function-schemata sub-module to HEAD (cf50aca)

Change #1144563 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (a293535)

https://gerrit.wikimedia.org/r/1144563

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/345

Update function-schemata sub-module to HEAD (a293535)

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/72

Update function-schemata sub-module to HEAD (a293535)

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/wikilambda-cli/-/merge_requests/72

Update function-schemata sub-module to HEAD (a293535)

Change #1144563 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (a293535)

https://gerrit.wikimedia.org/r/1144563

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/345

Update function-schemata sub-module to HEAD (a293535)

gengh merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/363

Minimize types as much as possible before sending to the evaluator.

Change #1145863 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Update evaluators from 2025-05-07-003410 to 2025-05-12-235119

https://gerrit.wikimedia.org/r/1145863

Change #1145864 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-05-06-142345 to 2025-05-14-112404

https://gerrit.wikimedia.org/r/1145864

Change #1145863 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Update evaluators from 2025-05-07-003410 to 2025-05-12-235119

https://gerrit.wikimedia.org/r/1145863

Change #1145864 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-05-06-142345 to 2025-05-14-112404

https://gerrit.wikimedia.org/r/1145864

I think we can close this. @DSantamaria , @ecarg , @gengh , is that accurate?

The linked reproduction step still has 500s in it; is that just caching?

In T393130#10822758, @Jdforrester-WMF wrote:

The linked reproduction step still has 500s in it; is that just caching?

I'm not sure! @gengh has more context on this one.

@gengh, can we close it?

This particular issue has been resolved.
Still some underlying failures (the 500s described by James), related to typed lists handling. This is already detected and being tracked in different tasks.

@DSantamaria @cmassaro @Jdforrester-WMF

The leftover errors behind https://www.wikifunctions.org/view/en/Z12864 are now being addressed in T394658: When running composition implementation of "lists have equal length", evaluator returns "Cannot read property .toString of undefined"