Thanks @gengh for pointing me to the call being sent to the orchestrator.

This is now passing on both of the linked tests. The error in the task description points to an executor error, and the empty string after it means that the executor just didn't return anything. Could have been something ephemeral like resource exhaustion or a timeout. Marking as "needs sign-off" for now.

I am gonna write some tests to "confirm" and then an MR to maybe fix. I can't truly replicate the problem locally, so we won't be 100% sure until the "fix" lands, but the fix shouldn't do harm either way.

I have an idea. I think what's happening here is that, when we populate the implementation list, we enter a race condition (due to an unfortunately-structured Promise.all).

Test call for ApiSandbox:

In T368892#9964816, @JMeybohm wrote:

Could you please elaborate what the fix is and why it fixes things?
Ah, missed the comment above about addNestedMetadata. But I still don't understand why this helps.

If I try running the function call above in the ApiSandbox, I get a "Service Unavailable."

Some of the failing tests also work if I use curl:

At least some calls to the orchestrator can run successfully with curl, and the response is basically instantaneous.

This is getting more relevant now. Image publication often fails and has to be re-run manually.

I've found more of these errors. They only seem to happen in compositions. This is making me think that maybe the result object is getting too big at some point in the execution. I'll dig into that.

We're still seeing the error after the above change. I agree that it's coming from the orchestrator (or at least the PHP -> orchestrator request), but the timeout we're getting is not the one implemented in the orchestrator's business logic. One of the Gateway Timeout errors we're getting doesn't even invoke the evaluator (https://www.wikifunctions.org/view/en/Z902), so evaluator latency doesn't explain everything.

This is now starting to mess things up. ZID resolution is interfering with our scope attachment (and therefore our reporting of nested metadata): we shouldn't have the dereferencer wrap returned objects in ZWrappers but leave that responsibility to the caller, which has the relevant correct scope information.

In /function-orchestrator/app.js, the system doesn't like req, res, or next, the params in app.all(). Anytime I try to log them, the logs get held up.

This has apparently been fixed for a while!

This will be an excellent experimental case for the propagation of granular error data once the nestedMetadata stuff is ready; I recommend we take this on AFTER nested metadata is done.

This was due to upstream changes in RustPython. @Jdforrester-WMF has since pinned our RustPython checkout to a specific tagged commit, so we shouldn't see this kind of volatility in future.

This is no longer true!

These tests are still failing. As far as I can tell, the "bad user-defined type" test in mswOrchestrateTest is redundant with "generic type validation error: bad list" in mockedServicesOrchestrateTest/genericsTest.js

ps-node also fails, in an even worse way: it kills the two parent processes but leaves the wasmedge process itself running (in a non-zombie state, even).

In T364889#9822313, @cmassaro wrote:

For what it's worth, I am not seeing this issue on Beta cluster

For what it's worth, I am not seeing this issue on Beta cluster

Whatever happened, it was ephemeral. I tried Wikifunctions.Debugging the arguments, and they were as expected. Removing the cached results and re-running caused the function to execute successfully.

I have replicated this locally. There are a few unfortunate things happening here. One is that our last-ditch effort (process.kill) still doesn't work in some cases, and every solution I've tried so far leaves us with zombie processes. It seems like the wasmedge environment does something that messes up process parent relationships, so I'll keep digging into that.

Very weird. When I test with the image that production is using, I can repro locally; however, when I use a more recent version of the orchestrator, I can run the function successfully. It's possible that the failure to recognize the Identity key, combined with the expansions of that key, might be responsible for the issue. In any case, I think this will resolve itself once we deploy a new version of the orchestrator on Monday, so we should retry then.