@GrounderUK @Jdforrester-WMF What's the meaning of "flagged as invalid results"?
The way I see this:
✅ The test is correctly marked as passing (green), because it passes (void === void)
✅ The metadata shows the errors received while evaluating the first call, which happen, and are expected.

What's the expected outcome of this task?

1. should the errors be hidden? I am not very convinced about this, because obfuscating real metadata returned by the orchestrator might make test case modeling much more complex.
2. should the errors returned by the orchestrator be presented or titled differently? this seems reasonable, but I'd love to hear some suggestions. What would be useful here?

“Flagged as invalid” means a Z560 was returned and interpreted as an “invalid execution result”. We don’t understand why this is “expected” from a valid call that evaluates correctly. What is the error and how can we avoid it?

In T404725#11213569, @gengh wrote:

[snip]

2. should the errors returned by the orchestrator be presented or titled differently? this seems reasonable, but I'd love to hear some suggestions. What would be useful here?

It would be useful to know what error the orchestrator thinks it has encountered. My premise is simply that, conceptually, there is no error here, so we don’t want to mislead people into thinking there is a problem that needs to be fixed. Because of T373607, we are not yet able to rely on a green tick.

Adding example function calls (because we can, thanks). Correctly echoing is no kind of error.

• Z560 echoing Unit-typed object
• echoing Z24/void

Pasting the result here for clarity:

{
  "Z1K1": {
    "Z1K1": "Z7",
    "Z7K1": "Z882",
    "Z882K1": "Z6",
    "Z882K2": "Z1"
  },
  "K1": "errors",
  "K2": {
    "Z1K1": "Z5",
    "Z5K1": "Z560",
    "Z5K2": {
      "Z1K1": {
        "Z1K1": "Z7",
        "Z7K1": "Z885",
        "Z885K1": "Z560"
      },
      "Z560K1": {
        "Z1K1": "Z99",
        "Z99K1": {
          "Z1K1": {
            "Z1K1": "Z9",
            "Z9K1": "Z22"
          },
          "Z22K1": {
            "Z1K1": {
              "Z1K1": "Z9",
              "Z9K1": "Z21"
            }
          }
        }
      }
    }
  }

Thanks. We don’t get to see the detail, but we do get a well-formed Unit, as far as I can tell.