The fix here is to use Kubernetes network policies to limit incoming traffic to the pygments server service.

That sounds fine, but surely there’s no reason to mount those secrets into the container in the first place? Then it wouldn’t matter who can access it.

Mentioned in SAL (#wikimedia-cloud) [2021-07-15T17:53:00Z] <wm-bot> <lucaswerkmeister> only mount ~/www/python into pygments-server pod (T286724)

In T286724#7215666, @LucasWerkmeister wrote:

That sounds fine, but surely there’s no reason to mount those secrets into the container in the first place? Then it wouldn’t matter who can access it.

That works too, as long as you make sure Kubernetes does not mount your service account to the pod on /var/run/secrets: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server. That also applies to function-evaluator. If you want to be extra safe you could do both.

Ugh, you’re right.

tools.lexeme-forms@interactive:~$ ~tools.notwikilambda/www/python/src/pygmentize /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -l sh </dev/null
-----BEGIN CERTIFICATE-----
REDACTED REDACTED REDACTED
-----END CERTIFICATE-----

Thanks.

Mentioned in SAL (#wikimedia-cloud) [2021-07-15T18:24:00Z] <wm-bot> <lucaswerkmeister> don’t mount service tokens into pods (T286724)

I thought a generally accessible pygments-server might potentially be useful, but now I don’t feel confident enough anymore that k8s isn’t mounting random stuff elsewhere in the container. Let’s see if I can figure out network policies.

Mentioned in SAL (#wikimedia-cloud) [2021-07-15T18:49:00Z] <wm-bot> <lucaswerkmeister> block external traffic to internal pods (T286724)

Alright, after a false start where I accidentally took down the wiki by blocking ingress to all pods, including the webservice, I think it’s working now.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: network-policy
  namespace: tool-notwikilambda
  labels:
    name: network-policy
spec:
  podSelector:
    matchLabels:
      networking: internal
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: tool-notwikilambda
    # reject all ingress connections from pods in other namespaces

(With networking: internal added to the spec.template.metadata.labels of the four deployments.)

The other fix is to do basic validation of the query string input before using it to access files on disk. This is OWASP 101 level stuff.

The whole point of my pygments-server, as opposed to the other one, is that it doesn’t parse the command line, because that limits the ways in which it can be used. I don’t control Extension:SyntaxHighlight, I don’t know which parts of pygments’ CLI it’ll start to use tomorrow.

Mentioned in SAL (#wikimedia-cloud) [2021-07-15T21:05:40Z] <wm-bot> <lucaswerkmeister> updated pygments-server to 2d4ab81a68 (prevent open(); T286724)

But as an approximation, I’ve pushed a pygments-server change to override the Python open() function before calling Pygments. It may break if Pygments changes how it reads files, but for now it works as another defense layer (I’ve tested it locally).