Could this be related to T377338 (?). I ran Z19232 on L1885, first time, and get the error reported in T377338 (gateway timeout). But the 2nd time and subsequent times I got the error reported here. Is it possible that "invalid json response body at ... wikilambda_fetch" was stored in the cache, due to the timeout of the first run?

This error can be reproduced by hittng the orchestrator on k8s-staging with this:

curl https://wikifunctions.k8s-staging.discovery.wmnet:30443/1/v1/evaluate/ -X POST --data '{"zobject":{"Z1K1":"Z7","Z7K1":"Z801","Z801K1":{"Z1K1":"Z7","Z7K1":"Z6825","Z6825K1":{"Z1K1":"Z6095","Z6095K1":"L47"}}},"doValidate":false}' --header "Content-type: application/json" -w "\n"

This call constructs a ZObject from lexeme L47, which is quite large, with many nested elements. Many nested elements cause there to be many calls from the orchestrator to wikilambda_fetch. (I counted 58 calls for processing L31815, and the number will be greater for processing L47.) we have a just-merged orchestrator code that greatly reduces the number of calls to wikilambda_fetch: MR 250

Is any reason to believe that too many calls to wikilambda_fetch could lead to this invalid json response body error? One next step is to deploy that patch and see what difference it makes.

The specific error we're getting is upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, which appears to be an error string coming from Envoy. This leads us to think that it's a networking issue (or something that's breaking the networking).

After the deployment of MR 250 and MR 251 today, the curl test above is working, but we get the same error by changing "L47" to "L1".

Currently, in my local environment, both L1 and L47 complete normally without errors. However, L1 still breaks on Wikifunctions.org.

This bug appears to be caused by a large number of nested elements inside the lexeme, which causes the resulting instance of Z6005/Wikidata lexeme to contain many different ZIDs, which in turn causes many calls to wikilambda_fetch API from the orchestrator. The problem was greatly alleviated by orchestrator patch MR 250 (link above). However, in the case of L1, there are 87 different language codes used in it, which results in 87+ different ZIDs in the resulting ZObject, so a minimum of 87 different calls to wikilambda_fetch.

L1-S1 has 87 different glosses (each of which uses a different language code) nested inside. The vast majority of senses have glosses in one or two languages. There are about ~300 that have 10+ glosses, and L1 is the one with the most (87), by far. The next one down has 49. If we remove the 100 largest, then we get down to 19 glosses. Here's the query for the top 1000.

The error returned has changed from invalid json response body at ... wikilambda_fetch to this:

"Z500K1":"Call tuples failed in returnOnFirstError. Error: Error: Added label "message" is not included in initial labelset: [ 'error', 'reqId' ].

(This Z500 message, as usual, is nested inside an instance of Z507.)

I'm changing the title of this ticket accordingly.

Hmm, I am trying this today, and I don't get this same error; this is what just came through.

However, I do get 'void' in the UI due to 'Error in Evaluation':

Hi @ecarg - Thanks for looking. Unfortunately, to see the complete error, it's currently necessary to open the Network tab (found in Developer tools, at least in Chrome), then run the function call, then click on the correct event in the Network tab, which is the one named just api.php, then scroll the Response until you can see the error. Currently, for me, it does still show up as the following (inside a larger error):

Call tuples failed in returnOnFirstError. Error: Error: Added label "message" is not included in initial labelset: [ 'error', 'reqId' ].

Showing the complete error in the metadata dialog is currently disabled because of its impact on performance, so you only get to see the top level there, which is Error in evaluation.