Change #1051364 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] Revert "wikifunctions: Upgrade orchestrator from 2024-06-11-223956 to 2024-06-17-221517"

https://gerrit.wikimedia.org/r/1051364

Change #1051364 merged by jenkins-bot:

[operations/deployment-charts@master] Revert "wikifunctions: Upgrade orchestrator from 2024-06-11-223956 to 2024-06-17-221517"

https://gerrit.wikimedia.org/r/1051364

Random evaluations:
22:00 UTC
✅https://www.wikifunctions.org/view/en/Z15386
✅https://www.wikifunctions.org/view/en/Z17105
❌https://www.wikifunctions.org/view/en/Z14226 Service Unavailable [passing at 22:50]
✅https://www.wikifunctions.org/view/en/Z10869
❌https://www.wikifunctions.org/view/en/Z11208 Service Unavailable [persistent]
✅https://www.wikifunctions.org/view/en/Z11684
22:30 UTC
✅https://www.wikifunctions.org/view/en/Z805 (equality function changed in one test)
✅https://www.wikifunctions.org/view/en/Z14304
23:00 UTC
❌https://www.wikifunctions.org/view/en/Z13633 Service Unavailable [passing at 00:30 UTC]
✅https://www.wikifunctions.org/wiki/Z808
⚠️https://www.wikifunctions.org/view/en/Z15251 Three Python implementations but only one passes (Z15377), otherwise Service Unavailable [Z15621 passing after 00:00 UTC; Z15252 has some passes too and is disconnected, probably because it fails for large values – errors are meaningful]
✅https://www.wikifunctions.org/view/en/Z17111 (with one odd exception: Z17117 with Z17114)

Change #1051813 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Raise CPU limit in orchestrator from 200m to 400m

https://gerrit.wikimedia.org/r/1051813

@cmassaro thanks for the ping. Unfortunately I have trouble finding the right data to analyze this further. The orchestrator does call the evaluator(s) directly (not via the service mesh) and does not produce metrics regarding those requests like latency, error rates etc. (at least I'm unable to find those). Looking at the logs of the components I can't see failed requests as well.

I think the "Gateway Timeout" message might be produced by the orchestrator (so the timeout happens while calling the evaluator) but as said I've no data really to back this up as I also I fail to understand where those errors are produced and if/how they bubble up the chain of services.

From the latency envoy telemetry metrics[1] of the evaluator I'd say that python function sometimes take pretty long (in the rage of 20-30s) which might make the call time out. This can't really be correlated with latency metrics from [2] as the bucket size for request latency seems to be 1s max (it might also make sense to split the dashboard or the panels between orchestrator and evaluators to get a clearer picture).
What does become pretty clear from the dashboard [1] is that the orchestrator and the python evaluator do have too low CPU limits. I'd suggest to double both as a first step and see how/if the throttled metric reduces.

[1] https://grafana-rw.wikimedia.org/d/b1jttnFMz/envoy-telemetry-k8s?orgId=1&var-datasource=thanos&var-site=eqiad&var-prometheus=k8s&var-app=function-evaluator&var-kubernetes_namespace=wikifunctions&var-destination=LOCAL_function-evaluator-javascript-evaluator&var-destination=LOCAL_function-evaluator-python-evaluator&from=now-2d&to=now
[2] https://grafana-rw.wikimedia.org/d/FEkiKFqVk/wikifunctions?orgId=1&from=now-6h&to=now