Page MenuHomePhabricator

Wikifunctions Kubernetes evaluator service exposes some sensitive data/functions within the container
Closed, ResolvedPublicSecurity

Description

I did some investigation of the exposed attack surface on the Wikifunctions evaluator Kubernetes service which is reachable by running Wikifunctions code, to help determine the impact of the recently reported issues by Lockal showing code was executable by users without the required permissions (T343263, T343138).

As expected the pod configuration (which is dedicated to Wikifunctions) is preventing access to other production services. The AppArmor policy also mitigates the ease with which host level attacks can be conducted. However there are some exposed sensitive data items accessible from within the evaluator running container environment that I would approximate at a MEDIUM risk:

  • Kubernetes service account secret is readable from wikifunctions code at /run/secrets/kubernetes.io/serviceaccount/token. The kubernetes master is not accessible over the network which means this token cannot be immediately used, but leaking it does make calls possible from other locations where it might be accessible (such as SSRF).
  • Envoy admin interface is reachable at http://127.0.0.1:1666. This allows access to the running configuration (for example by reading /config_dump endpoint) although private keys are redacted in the output. It also allows some configuration changes. According to the documentation it is critical that access to the administration interface is only allowed via a secure network.
  • An endpoint is accessible over localhost port 9361. I believe this is the Istio telemetry API although I've not been able to confirm that yet. In which case this would allow obtaining access logs, metrics and traces, as well as configuring of them. If so this probably isn't advisable to be exposed to Wikifunctions code.

In addition to the above there are currently some improvements that could be made to the AppArmor policy and other defense in depth measures. It would be nice to have a roadmap for gradually improving the lockdown of the containers. Given my own experience I thought it may help to point out some of the things that made attack surface analysis easier:

  • /proc/net/tcp is exposing the container-wide networking TCP socket state for all processes. I would recommend using the AppArmor policy to prevent read access to the unneeded /proc, /run and /sys filesystem entries. This would not prevent attacks but it would make the attack surface enumeration harder. (Accessing the container binaries are not a concern
  • The accessibility of network sockets running over the loopback interface could also be restricted via AppArmor policy to prevent unwanted connections. This would be a good mitigation for some of the attacks above.
  • The ability to execute shell commands could also be restricted via AppArmor policy. It would prevent some more advanced system profiling than is easily possible from within the supporting programming environments (such as Linux capability system).
  • Extending the use of gvisor to production would decrease the likelihood of a host escape (although preventing the writing of files is making an exploit much harder as its not possible to run an arbitrary ELF binary, and the limited permissions of the executor user is also decreasing the syscall attack surface significantly so it's not as urgent as it would otherwise be).

Event Timeline

The current AppArmor profile for the evaluator denies a bunch of specific /proc and /sys reads (and more writes), but not a wholesale ban. @cmassaro, @JMeybohm, did you have particular reasons for those restrictions? Were they generated by hand or via bane or similar?

Jdforrester-WMF moved this task from To triage to Backlog on the Abstract Wikipedia team board.

The current AppArmor profile for the evaluator denies a bunch of specific /proc and /sys reads (and more writes), but not a wholesale ban. @cmassaro, @JMeybohm, did you have particular reasons for those restrictions? Were they generated by hand or via bane or similar?

These are from the defaults we do apply to all containers (see https://gerrit.wikimedia.org/r/c/operations/puppet/+/940152/15/modules/profile/files/kubernetes/node/wikifunctions-evaluator#19).
Ideally the profile should deny all reads as well and just include the appropriate abstractions for whatever you run (as seen in the examples in T326785, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/940152/15/modules/profile/files/kubernetes/node/wikifunctions-evaluator#14) or better declare the files it needs read access to directly in the profile (as the abstraction might change with the OS of the kubernetes node).

Kubernetes service account secret is readable from wikifunctions code at /run/secrets/kubernetes.io/serviceaccount/token. The kubernetes master is not accessible over the network which means this token cannot be immediately used, but leaking it does make calls possible from other locations where it might be accessible (such as SSRF).

We should disable auto mounting of the service account as described in: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#opt-out-of-api-credential-automounting

DVrandecic changed the task status from Open to In Progress.Aug 14 2023, 3:41 PM
DVrandecic moved this task from Backlog to In Progress on the Abstract Wikipedia team board.
@acooper wrote:
  1. Kubernetes service account secret is readable from wikifunctions code at /run/secrets/kubernetes.io/serviceaccount/token. The kubernetes master is not accessible over the network which means this token cannot be immediately used, but leaking it does make calls possible from other locations where it might be accessible (such as SSRF).
  2. Envoy admin interface is reachable at http://127.0.0.1:1666. This allows access to the running configuration (for example by reading /config_dump endpoint) although private keys are redacted in the output. It also allows some configuration changes. According to the documentation it is critical that access to the administration interface is only allowed via a secure network.
  3. An endpoint is accessible over localhost port 9361. I believe this is the Istio telemetry API although I've not been able to confirm that yet. In which case this would allow obtaining access logs, metrics and traces, as well as configuring of them. If so this probably isn't advisable to be exposed to Wikifunctions code.

I've fixed the first two points with some change to our service mesh config and https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/953291. tcp/9361 is where we expose the local envoy health check and prometheus metrics, nothing else is allowed on that port.

sbassett changed Author Affiliation from N/A to WMF Product.Nov 8 2023, 5:15 PM
sbassett changed the visibility from "Custom Policy" to "Public (No Login Required)".
sbassett changed the edit policy from "Custom Policy" to "All Users".
sbassett changed Risk Rating from N/A to High.