Verified as fixed in beta.

Verified in beta as fixed.

This is a bug in the Orchestrator's validation logic. Suppose we have:

De-assigning this task from myself, as my fellowship comes to an end. Will leave it to the team to close this ticket as appropriate.

Echo on this object (which is identical to Z10486, except the list has no elements) succeeds: https://wikifunctions.beta.wmflabs.org/wiki/Z10558
Echo on this object (which is identical to the above but with an element) fails: https://wikifunctions.beta.wmflabs.org/wiki/Z10559

This task is filed as 'nice-to-have', but as it stands the table is pretty broken, and I don't think it should be launched in its current state. Could we remove it for now, then restore it in a working fashion post-launch?

Re-opening this to make sure the documentation here and here is updated. Now that the script is gone, as far as I can tell one must manually update dependencies.json (optionally copy-pasting from the output of the test), but I may be missing something.

This is passing consistently when I run it today.

Fixed by Dani's https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/866449.

Waiting on comment from team on this doc, presenting two options for a list equality function.

Waiting on comment from wider team on https://docs.google.com/document/d/1rUK1zZbiTuy025XPJbdl9xvt_07UVL0b4xRDZovu-Ok.

I've updated the documentation at https://www.mediawiki.org/wiki/Extension:WikiLambda/API#wikilambda_perform_test. I think we can now close this ticket.

Regarding the remaining tasks:

We agreed in the AW Engineering meeting that we would create a predefined List Equality function, with a built-in implementation, to use in this test.

The issue with Z10066 was T324845, which I am investigating. I've fixed Z10066 for now by removing it's non-English labels.

Z10061 has two testers:

• Z10340: this is not well-formed - the outer function in its Z20K2 does not match Z20K1. This is required so that the PHP layer can pass in a version of the function containing the implementation under test.
• Z10066: this is failing with "Duplicate Z11K1/language element in Z12/Multilingual text: 'undefined'" - looking into it.

This may be due to Z10340 not being well-formed - the outer function in Z20K2 is required to match the function in Z20K1.

Re: the actual topic of this ticket (Z8013 not passing), the above discussion is for now a moot point. The test is failing before the validation step is ever reached. I'm looking into this.

In T314621#8450985, @EWright-WMF wrote:

The responseObject returned by the Orchestrator has changed since this ticket was filed. The current problem with it is detailed in T324661.

Thanks. Filed T324687 for the PHP issue and will tackle that now. Will leave the Orchestrator issue here for later triage.

In T324661#8451119, @Jdforrester-WMF wrote:

In T324661#8450386, @EWright-WMF wrote:

P.S. My assumption here is that it is the Orchestrator that is wrong, rather than the PHP layer being wrong in thinking the response is malformed. Happy to be corrected.

I think the orchestrator is correct and the PHP is wrong (this is well-formed; types in Benjamin arrays can be calls or inline/dereferenced types, not just references), but we're not meant to be getting them in this case – it should indeed be Z6.

The responseObject returned by the Orchestrator has changed since this ticket was filed. The current problem with it is detailed in T324661.

P.S. My assumption here is that it is the Orchestrator that is wrong, rather than the PHP layer being wrong in thinking the response is malformed. Happy to be corrected.

As James was (I think) suggesting, we don't actually strictly need a built-in implementation to achieve a list equality function. As a POC, I created a new function in beta, 'List of Strings equality', and a recursive composition implementation: https://wikifunctions.beta.wmflabs.org/wiki/Z10338. This function and implementation are specifically for testing equality of List<String>, but you could make a more general list equality function by adding another argument for an equality function, and then calling this when comparing individual elements. I tried doing this, but there are some (I think) bugs preventing setting Z7K1 to an argument reference (T324600, T324597).

In T299194#8261275, @Jdforrester-WMF wrote:

In T299194#8261245, @cmassaro wrote:

Should we add List equality as a built-in, then? Or just remove this test?

Can we recurse through the list with a Map() applying the StringEquality()?

Thanks - I'll work on that now.

Also, would you mind clarifying what is meant by the item "Response was a failure"? If you mean just that the test fails (returns false), several of the existing test cases cover this.

Thanks James for updating the checklist. I think also " Running on an unknown ZObject as tester" is done - see L242: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikiLambda/+/refs/heads/master/tests/phpunit/integration/API/ApiPerformTestTest.php#242. I've updated the checklist.

@Jdforrester-WMF - apologies, I didn't mean to bypass processes. I've removed the tag and will leave it to to be triaged.

(adding back onto Phase Theta board to reflect conversation at last Experience Planning meeting, where it was agreed I would prioritise this)

Hi @Jdforrester-WMF - I agree this probably just needs documentation updating, which I can take on.

Hi @DMartin-WMF - I can take a look at that. But given it's just one of the tests that's failing I suspect it's an issue with function orchestration or evaluation, rather than the perform test flow itself.

The above patch ensured both implementations and testers are converted to JSON.