Page MenuHomePhabricator
Create Task
Maniphest T428653

Orchestrator error logs from Monday 2026-06-08: Don't throw 'unidentified error' on evaluator 4xx responses
Closed, ResolvedPublic
Actions

Description

one occurrence of:

Orchestration failed with unidentified error: Error: Function evaluation failed with status 400: Failed to parse the request body as JSON: functionArguments.Z21580K1.object.Z6001K2.object.Z12K1.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K1.object.Z11K1.object.Z60K1.rawObject: recursion limit exceeded at line 1 column 11758.
  • I couldn't find the incoming request for this request-id! ed1d5d78-b109-4afe-907f-0a86805b29f5

one occurrence of:

Orchestration failed with unidentified error: Error: Function evaluation failed with status 400: Failed to parse the request body as JSON: functionArguments.Z30216K1.object.Z6005K7.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K1.object.Z6004K4.object.K2.object.K2.object.K2.object.K1.object.Z6091K1.rawObject: recursion limit exceeded at line 1 column 66190.
  • none of the related Orch request data produced this error locally

one occurrence of:

	Orchestration failed with unidentified error: Error: Function evaluation failed with status 400: Failed to parse the request body as JSON: functionArguments.Z22853K1.object.Z6001K2.object.Z12K1.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K2.object.K1.object.Z11K1.object.Z60K1.rawObject: recursion limit exceeded at line 1 column 12014.
  • also not able to reproduce locally

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
wikifunctions: Upgrade orchestrator from 2026-06-11-171152 to 2026-06-16-183209Jforresteroperations/deployment-chartsmaster+1 -1
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
orchestrator: surface evaluator 4xx responses as typed Z507 rather than 'unidentified error'repos/abstract-wiki/wikifunctions/function-orchestrator!717ecarggrace/T428653/handle-json-parse-errormain
Customize query in GitLab

Related Objects

Event Timeline

ecarg created this task.Jun 9 2026, 5:32 PM
ecarg updated the task description. (Show Details)
ecarg updated the task description. (Show Details)Jun 9 2026, 5:35 PM
ecarg updated the task description. (Show Details)Jun 9 2026, 5:40 PM
Jdforrester-WMF subscribed.Jun 9 2026, 5:42 PM

Sounds like we need to add a recursion limit guard in the evaluator somewhere and throw an appropriate Z5xx, but it's not clear where. A stack trace would be so helpful. :-)

Jdforrester-WMF added a comment.Jun 9 2026, 7:32 PM

Wait, these errors are both coming from the orchestrator, not the evaluator. Are you sure this should block the Rust evaluator roll-out?

Jdforrester-WMF removed a project: Epic.Jun 10 2026, 5:28 PM
cmassaro added a comment.Jun 11 2026, 3:24 PM

This isn't Rust's fault. It's this: https://phabricator.wikimedia.org/T428917

ecarg added a comment.Jun 12 2026, 3:39 PM

I was told to file the log errors from my Chores under this task, but I'm just now getting to the repro steps and agree, this doesn't seem related to the Rust Eval!

ecarg renamed this task from [rust eval] Failed to parse the request body as JSON logs to Failed to parse the request body as JSON logs.Jun 12 2026, 3:41 PM
ecarg edited projects, added: function-orchestrator; removed: function-evaluator.
Jdforrester-WMF removed a parent task: T421849: [Hypothesis] WE2.3.16 Productionize Rust Evaluator and Deprecate Node Evaluator.Jun 12 2026, 4:04 PM
ecarg renamed this task from Failed to parse the request body as JSON logs to error logs from Monday 260608.Jun 12 2026, 5:02 PM
ecarg changed the task status from Open to In Progress.
ecarg triaged this task as Medium priority.
ecarg updated the task description. (Show Details)
ecarg updated the task description. (Show Details)Jun 12 2026, 5:14 PM
CodeReviewBot added a project: Patch-For-Review.Jun 12 2026, 5:45 PM

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/717

Draft: throw exception when json parse error

Jdforrester-WMF edited projects, added: Essential-Work; removed: Patch-For-Review, OKR-Work.Jun 12 2026, 10:10 PM
ecarg claimed this task.Mon, Jun 15, 9:06 AM
ecarg moved this task from Incoming to In Code review on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.
CodeReviewBot added a comment.Mon, Jun 15, 9:56 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/717

orchestrator: surface evaluator 4xx responses as typed Z507 rather than 'unidentified error'

ecarg moved this task from In Code review to Ready to deploy on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.Mon, Jun 15, 10:12 PM
Jdforrester-WMF renamed this task from error logs from Monday 260608 to Orchestrator error logs from Monday 2026-06-08: Don't throw 'unidentified error' on evaluator 4xx responses.Mon, Jun 15, 11:05 PM
gerritbot added a comment.Wed, Jun 17, 1:49 PM

Change #1303434 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2026-06-11-171152 to 2026-06-16-183209

https://gerrit.wikimedia.org/r/1303434

gerritbot added a project: Patch-For-Review.Wed, Jun 17, 1:49 PM
gerritbot added a comment.Wed, Jun 17, 9:19 PM

Change #1303434 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2026-06-11-171152 to 2026-06-16-183209

https://gerrit.wikimedia.org/r/1303434

Jdforrester-WMF moved this task from Ready to deploy to Needs Sign-off on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.Wed, Jun 17, 9:22 PM
ecarg closed this task as Resolved.Thu, Jun 18, 2:34 PM

Handling these, Orchestration failed with unidentified error: Error: Function evaluation failed with status 400: "could not evaluate successfully is resolved. Now seeing a bunch of the same except with 5xxs -_-; I can extend the condition added to cover these as well.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits