Because the evaluator runs untrusted user code, it currently isn't allowed to open network connections to anywhere. In order to let user code make calls to other wikifunctions, we're now loosening that policy in a limited way. In particular, only a limited set of functions are allowed (none of which in turn are implemented in user code). That restriction is enforced at a few levels:
- The orchestrator grows a new HTTP endpoint /v1/callback with the same semantics as the normal evaluation entry point, except that it enforces this function allowlist, along with other limitations on the input format.
- The orchestrator's Envoy sidecar grows a second inbound TLS port, which routes the URL /v1/callback to the same local upstream (i.e. the same orchestrator instance) and emits a 403 for everything else.
- The evaluator's Envoy sidecar grows a route to the limited port on the orchestrator. (The URL limitation needs to be on the orchestrator side, though, because a compromised evaluator could just skip the local Envoy.)
- The evaluator's egress network policy (previously "nothing to nobody") grows a rule allowing traffic to the limited port on the orchestrator.
#1 is another part of the parent task; the rest are in scope here. #2 is a straightforward Envoy listener definition, and #3 and #4 are more interesting.
The evaluator-to-orchestrator callback traffic can't go through ingress; we'd lose port isolation. It also doesn't really need to go through LVS, since it might as well stay intra-DC, and standing up a whole new LVS service to reach the same backend would incur a lot more overhead complexity than we need. My current thinking is for the evaluator Envoy's upstream to be the orchestrator's Kubernetes svc.cluster.local address, which is nonstandard but buys us into just the right level of abstraction here.
That in turn means doing a little config finagling, because right now our whole service mesh setup assumes that Envoy listeners are 1:1 with LVS services. We could extend that at the mesh.configuration level, to add a values switch for intra-cluster traffic, but I don't think that's actually the right call; my instinct is that the reusability of this is pretty low. Instead, we'll inject a minimal services_proxy dict into the evaluator's values, overriding the cluster-wide one from helmfile-defaults, containing the cluster.local address and dedicated port number, but otherwise duplicating the config generated in Puppet. (That introduces some risk of drift from what's in Puppet. It doesn't change often, at least. Someday, this can be a custom codepath in Sophroid keyed on the Node info, which would make it easier to keep things in sync.)
Finally for the network policy, we can add an extraRules the same as the orchestrator's, but pointed back the other way, which should clear us of any DNAT-related issues.
I have a stack of patches in progress for this, will upload today or tomorrow.