Page MenuHomePhabricator

Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection
Closed, ResolvedPublic

Description

In the Cloud VPS OpenStack environment we have an account named "novaobserver" that is associated with a role named "observer" which we add to every Cloud VPS project (T150092). The custom "observer" role is defined in our OpenStack configuration to allow it to inspect, but not change, a wide variety of OpenStack configuration related to each Cloud VPS project. The most visible use of this shared account is https://tools.wmflabs.org/openstack-browser/ which uses it extensively to interrogate the state of the OpenStack deployment.

We would like tools to be able to do similar inspection of cluster and namespace state in the Toolforge Kubernetes cluster. T201892: Toolforge: Build dashboard that breaks down webservice type usage by tool is one concrete example of a desired tool. Kubernetes equivalents of https://tools.wmflabs.org/sge-jobs/ and https://tools.wmflabs.org/sge-status/ would be other desired tools.

The "k8sobserver" role/user/whatever should NOT be able to see "private" or "secret" things in a namespace. This certainly includes [[https://kubernetes.io/docs/concepts/configuration/secret/|Secret]] objects, and also probably should extend to ConfigMap objects. The approach taken with the novaobserver account is an allowlist rather than a blocklist so that we do not accidentally expose new things before we have made a reasoned examination of their security/privacy implications.

Related Objects

Event Timeline

bd808 created this task.Sep 19 2019, 11:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2019, 11:45 PM
Bstorm added a subscriber: Bstorm.Sep 19 2019, 11:54 PM

Oddly, since most serviceaccounts are namespaced, this may be easier to do with a simple "user" object with an x509 and a custom role that gives just the perms you need. We'll think more about it...

The "k8sobserver" role/user/whatever should NOT be able to see "private" or "secret" things in a namespace. This certainly includes Secret objects, and also probably should extend to ConfigMap objects.

I wonder how many ConfigMaps we actually have, depending on that maybe we could review them to check whether there's anything secret stored in there. We could have maintainers move secrets to Secret objects and then open up ConfigMap access.

Krenair added a comment.EditedSep 22 2019, 12:08 AM

The "k8sobserver" role/user/whatever should NOT be able to see "private" or "secret" things in a namespace. This certainly includes Secret objects, and also probably should extend to ConfigMap objects.

I wonder how many ConfigMaps we actually have, depending on that maybe we could review them to check whether there's anything secret stored in there. We could have maintainers move secrets to Secret objects and then open up ConfigMap access.

So I'm entirely new to this k8s cluster but, based on kubectl get --all-namespaces cm on tools-k8s-master-01, it looks like the answer is 2, and it looks like their contents would be fine to expose.

Bstorm added a comment.Oct 4 2019, 8:38 PM

On the new cluster, config maps are likely to serve additional purposes. Nobody SHOULD use configmaps for private or secret data. That doesn't guarantee they won't, but they shouldn't. There will be a configmap for each user at a minimum that will include the expiration date of their user certs (and is part of the instrumentation of their account).

The existing cluster on tools-k8s-master-01 is only passingly similar to the upgraded one in general. Tools that use the webservice command will get much of what they get now, with the addition of a bunch of other things.

If nothing else, perhaps this should clarify that it is for the new cluster. On the current cluster, I'd just make an admin user with a static token with a hand crafted ABAC thing.

That said, whether this is a service account in every namespace that the main account assumed the identity of or just a user with "cluster read", there's a default clusterrole called view in any recent version of k8s with rbac enabled that looks like this:

$ kubectl get clusterrole view -o yaml
aggregationRule:
  clusterRoleSelectors:
  - matchLabels:
      rbac.authorization.k8s.io/aggregate-to-view: "true"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2019-09-18T18:22:25Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
  name: view
  resourceVersion: "346"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/view
  uid: 6c101b9c-e546-4b70-a3af-e4168180d497
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - persistentvolumeclaims
  - pods
  - replicationcontrollers
  - replicationcontrollers/scale
  - serviceaccounts
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - bindings
  - events
  - limitranges
  - namespaces/status
  - pods/log
  - pods/status
  - replicationcontrollers/status
  - resourcequotas
  - resourcequotas/status
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - controllerrevisions
  - daemonsets
  - deployments
  - deployments/scale
  - replicasets
  - replicasets/scale
  - statefulsets
  - statefulsets/scale
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - deployments/scale
  - ingresses
  - networkpolicies
  - replicasets
  - replicasets/scale
  - replicationcontrollers/scale
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  - networkpolicies
  verbs:
  - get
  - list
  - watch

Basically, it sees all, and that's it. The only secret it would be able to view is its own credentialing one if it is using service accounts. If it is connected via a clusterrolebinding, then it can see all in all namespaces as well.

Bstorm added a comment.Oct 4 2019, 8:39 PM

Heh, in fact on the current cluster, while the user is easy, deploying anything is hard because there are no exceptions to the admission controllers. The new one will be more flexible.

bd808 triaged this task as Medium priority.Dec 18 2019, 6:51 PM

On the new cluster, config maps are likely to serve additional purposes. Nobody SHOULD use configmaps for private or secret data. That doesn't guarantee they won't, but they shouldn't. There will be a configmap for each user at a minimum that will include the expiration date of their user certs (and is part of the instrumentation of their account).

The existing cluster on tools-k8s-master-01 is only passingly similar to the upgraded one in general. Tools that use the webservice command will get much of what they get now, but they will need a bit more in every case...plus the new cluster will have RBAC at all.

I am also hoping to do fewer host mounts and more config maps for most of the rest of the data exposed to pods (later edit after coming back to this: nope not right now, lol)

Overall, I do think that the default 'view' clusterrole is suitable, partly because we are likely to want to be able to track new services. Just need a quick script or doc to enable an appropriate service account with that role in a tool namespace on request.

Bstorm claimed this task.Dec 18 2019, 7:04 PM
Bstorm moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

@bd808 Do you have a tool you'd like to include in this initially? That would make it easier for me, honestly, to reason through a script or process to enable it.

Writing up a script that can be dropped on the control plane nodes that will allow any arbitrary tool to be granted a service account that has access to the view role on request.

bd808 added a comment.Dec 18 2019, 8:33 PM

@bd808 Do you have a tool you'd like to include in this initially? That would make it easier for me, honestly, to reason through a script or process to enable it.

The k8s-status tool would be a good candidate. That's where I would like to build some kind of "what is active on the Toolforge k8s cluster" website.

Almost done with a script to test. Just need to set up a read-only PSP that we can grant it across the cluster as well as RBAC.

It might be sufficient to use the tool's psp

Ok I have a script that works nicely. The only thing I kind of dislike is that it can see all pods and configmaps in kube-system. That's not really a problem, I suppose. None of that is secret (in fact, most is effectively in puppet). I'm sure some places would want to rope that off, though :)

Example below from a pod created with the serviceaccount specified, using any old image, I downloaded kubectl to the /tmp directory and ran it (kubectl automagically looks for the service account creds).

$ ls
kubectl
$ ./kubectl get pods --all-namespaces
NAMESPACE            NAME                                                   READY   STATUS    RESTARTS   AGE
ingress-admission    ingress-admission-55fb8554b5-j9xds                     1/1     Running   1          6d21h
ingress-admission    ingress-admission-55fb8554b5-wmr46                     1/1     Running   0          6d21h
ingress-nginx        nginx-ingress-64dc7c9c57-jkn75                         1/1     Running   1          6d2h
ingress-nginx        nginx-ingress-64dc7c9c57-kpkk4                         1/1     Running   0          6d2h
ingress-nginx        nginx-ingress-64dc7c9c57-nzm76                         1/1     Running   0          6d2h
kube-system          calico-kube-controllers-59f54d6bbc-dwg76               1/1     Running   1          15d
kube-system          calico-node-2sf9d                                      1/1     Running   5          56d
kube-system          calico-node-dfbqd                                      1/1     Running   1          56d
kube-system          calico-node-g4hr7                                      1/1     Running   2          56d
kube-system          calico-node-lp2c9                                      1/1     Running   0          7d22h

So any tool that I grant this to can use a particular service account to see into almost everything. It cannot run with any greater priv escalation than your usual account and actually has less read/write powers (basically none). Webservice doesn't allow injecting a service account yet (but this is a good future feature in a way), so a start/stop script that reads yaml sounds like the way to go.

Patch incoming...

Change 559212 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: add a script to grant "observer" access to a tool

https://gerrit.wikimedia.org/r/559212

bd808 added a comment.Dec 18 2019, 9:40 PM

Webservice doesn't allow injecting a service account yet (but this is a good future feature in a way), so a start/stop script that reads yaml sounds like the way to go.

I imagine the actual tool I would build would be a flask app using the k8s python client library, so I don't think that webservice actually needs anything fancy? Or is it somehow not possible to call back to the k8s API server using different credentials than the ones that the pod running the code is using?

Bstorm added a comment.EditedDec 18 2019, 9:57 PM

It does. Unless I change that script to grant the "default" service account (that's it's actual name) for your namespace these privileges, then you need to specify a service account on launch of the pod or more likely deployment managing the pod.

I can provide a template or script for that, and it'll make perfect sense when you see it, if needed. Everything has a service account, and not specifying one gives you the "default" one, which maintain-kubeusers gives the rights of a tools user (so that webservice-created replicasets get the PSP needed to start pods otherwise, that service account can only do basic pod ops). It's considered sort of a bad practice to use the default SA in a pod since the SA defines the perms of a service within k8s (like how an x509 defines a human or outside user's perms), but it is "ok", so I left that alone in webservice and worked around it because it would be a pain to introduce to all users. That's part of that whole, "don't force your devs to directly interact with kubernetes a lot" notion. I figure we can adapt our tooling more to that idea once we are entirely on a cluster that supports all the APIs at least.

So yeah, I should say that I *can* just give the default service account of the namespace full cluster read-only (allowing webservice to work as usual), but I don't *want* to. I'd rather just provide a start/stop script that sets the SA in the few tools that need this, if it's all the same to you. That also makes it easier to remove the perms if said tool stops needing it.

Here's a nice script you could use to launch...you'll notice two things different here than what you get from webservice. For one, the deployment is in api group apps/v1, that means you don't have to do all that label-searching because cascading deletes work (thus my frustration with pykube). Second, the service account is set in the template for the deployment. That and it merely suggests what to do to restart the app instead of doing what webservice does to take some strain off the ingress controllers.

It sets the toolforge: tool label because that's what mounts your NFS dirs, SSSD stuff and sets your home env var.

1#!/bin/bash
2
3# This is intended to be run as a tool user
4set -Eeuo pipefail
5
6function usage {
7 echo -e "Usage (must be toolforge user account):\n"
8 echo "k8s_webservice.sh <start|stop>"
9 echo ""
10 echo "Example: k8s_webservice.sh start"
11}
12
13function startsvc {
14 echo "starting..."
15 cat <<EOF | kubectl apply -f -
16apiVersion: apps/v1
17kind: Deployment
18metadata:
19 labels:
20 name: $1
21 toolforge: tool
22 tool.toolforge.org/service: "true"
23 name: $1
24spec:
25 replicas: 1
26 selector:
27 matchLabels:
28 name: $1
29 toolforge: tool
30 tool.toolforge.org/service: "true"
31 template:
32 metadata:
33 labels:
34 name: $1
35 toolforge: tool
36 tool.toolforge.org/service: "true"
37 spec:
38 serviceAccountName: ${1}-obs
39 containers:
40 - command:
41 - /usr/bin/webservice-runner
42 - --type
43 - uwsgi-python
44 - --port
45 - "8000"
46 image: docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web:latest
47 imagePullPolicy: Always
48 name: webservice
49 ports:
50 - containerPort: 8000
51 name: http
52 protocol: TCP
53 workingDir: /data/project/${1}/
54EOF
55
56 cat <<EOF | kubectl apply -f -
57apiVersion: v1
58kind: Service
59metadata:
60 labels:
61 name: $1
62 toolforge: tool
63 tool.toolforge.org/service: "true"
64 name: $1
65spec:
66 ports:
67 - name: http
68 port: 8000
69 protocol: TCP
70 targetPort: 8000
71 selector:
72 name: $1
73 type: ClusterIP
74EOF
75
76 cat <<EOF | kubectl apply -f -
77apiVersion: networking.k8s.io/v1beta1
78kind: Ingress
79metadata:
80 annotations:
81 nginx.ingress.kubernetes.io/rewrite-target: /$1/\$2
82 labels:
83 name: $1
84 toolforge: tool
85 tool.toolforge.org/service: "true"
86 name: $1
87spec:
88 rules:
89 - host: tools.wmflabs.org
90 http:
91 paths:
92 - backend:
93 serviceName: $1
94 servicePort: 8000
95 path: /$1(/|$)(.*)
96EOF
97}
98
99function stopsvc {
100 echo "stopping..."
101 echo "First the ingress"
102 kubectl delete ingress $1
103 echo "Now the service"
104 kubectl delete svc $1
105 echo "And the actual app"
106 kubectl delete deployment $1
107}
108
109wmcsproject=$(</etc/wmcs-project)
110if ! [[ $USER == "${wmcsproject}."* ]]; then
111 printf >&2 '%s: user name does not start with "%s": %s\n' "$0" "$wmcsproject" "$USER"
112 usage
113 exit 1
114fi
115
116prefix=$(($(echo -n $wmcsproject | wc -c)+1))
117
118tool="${USER:prefix}"
119
120case $1 in
121 start)
122 startsvc "$tool"
123 ;;
124 stop)
125 stopsvc "$tool"
126 ;;
127 restart)
128 echo "For a restart, this seems kind of heavy"
129 echo "Why don't you just run \"kubectl get pods\" and the \"kubectl delete pods <podname>\"?"
130 ;;
131 *)
132 echo "There must be an argument of start or stop"
133 usage
134 exit 1
135 ;;
136esac
137

The basic idea is that this clearly introduces the need for a serviceaccount field in webservice, unless we change tooling a lot.

Change 559212 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: add a script to grant "observer" access to a tool

https://gerrit.wikimedia.org/r/559212

If this process works for you for now (we can probably hack in a service-account setting for webservice if we ever want to...but the API versions I used in the bash script are actually likely to work slightly better because they aren't deprecated), I can run the script to give the service account to the tool namespace any time.

@bd808
You should be clear to start fiddling with k8s-status on the new cluster when you migrate it now:

root@tools-k8s-control-1:~# wmcs-k8s-enable-cluster-monitor k8s-status
NAME              STATUS   AGE
tool-k8s-status   Active   2d17h
Creating the service account...
serviceaccount/k8s-status-obs created
Enabling read-only access to the cluster...
clusterrolebinding.rbac.authorization.k8s.io/k8s-status-obs created
rolebinding.rbac.authorization.k8s.io/k8s-status-obs-psp created
NAME             SECRETS   AGE
k8s-status-obs   1         1s
*********************
Done!

I'd just use P9946 or something similar for starting and stopping the service.

Bstorm closed this task as Resolved.Dec 19 2019, 7:34 PM

I think this is basically done then.