Update: with d8b9cf42cc17bfb2988ae0d32c7b8efea1565d19 in the orchestrator (and the equivalent in the evaluator), this is now showing up as:

Otherwise Unhandled error 500: {"status":500,"type":"internal_error","detail":"Expected ',' or '}' after property value in JSON at position 413"}

… which suggests we're failing to try/catch a JSON decode from user input (?) at some point.

• builtins.getLanguageMap – but should be very safe
• Evaluator.evaluateReentrant_ - only in re-entrancy, not enabled
• fetchObject.fetchZIDs - suspect, though should be safe
• fetchObject.retrieveWikidataEntityFromLODAPI - suspect, though should generally be safe (we check for 404s/etc.)
• fetchObject.findEntitiesByStatements - suspect, though should should generally be safe (we check for 404s/etc.)
• utils.getEvaluatorAndOrchestratorConfigsForEnvironment – should either be entirely safe or explode on every request
• utils.readJSON
  • builtins.getPersistentZObjectFromFile & …getDefinitionFromFile – should be safe
  • invariants.softwareLanguages – should either be entirely safe or explode on every request

Consequently I'm most suspicious of the fetchZIDs one, but we should (a) try/catch and re-error as appropriate with a trace, and (b) consolidate on using a wrapper that does this.

apine opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/310

Wrap fetchZIDs's JSON.parse in try{} and test that.

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/310

Wrap fetchZIDs's JSON.parse in try/catch.

Change #1129368 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-03-11-234105 to 2025-03-19-203723

https://gerrit.wikimedia.org/r/1129368

Change #1129368 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Update orchestrator from 2025-03-11-234105 to 2025-03-19-203723

https://gerrit.wikimedia.org/r/1129368

OK, this should now fail with wikilambda_fetch returned degenerate output, which will help us find out where these are coming from.

noting here that while looking at logs, latest error message object is 'request entity too large' which seems like a Node.js error

As far as I can tell, this error (which now manifests as "Otherwise Unhandled error [...]" ... doesn't prevent the function call from running? If I filter by the request ID for a request that causes this error, the error happens before orchestration even begins--and yet orchestration continues unimpeded. So I think this error is not very informative and can be ignored.

@cmassaro can we close this?

@ecarg, did you mean to close it? (You moved it to open, not sure why)

@cmassaro, is this resolved?

In T388203#10833367, @DSantamaria wrote:

@cmassaro, is this resolved?

From my perspective, the status remains as discussed before:

In T388203#10705731, @cmassaro wrote:

As far as I can tell, this error (which now manifests as "Otherwise Unhandled error [...]" ... doesn't prevent the function call from running? If I filter by the request ID for a request that causes this error, the error happens before orchestration even begins--and yet orchestration continues unimpeded. So I think this error is not very informative and can be ignored.

So yes, I think we can resolve it for now.