Page MenuHomePhabricator

Design and document how to integrate the new Toolforge k8s cluster with PodSecurityPolicy
Closed, ResolvedPublic

Description

The PodSecurityPolicy is something we really need for the security basics of running pods in the cluster.

Related docs: https://kubernetes.io/docs/concepts/policy/pod-security-policy/

NOTE: If this option is enabled but left unconfigured at cluster bootstrap time, the cluster won't bootstrap at all (basic control plane pods won't be created).

Since Pod Security Policies (PSPs) are tightly integrated with RBAC, this task must address that as well.
It seems that so far, placing the PSPs with accompanying RBAC in an /etc/kubenetes subdir is agreed upon as well as likely combining everything needed to launch the cluster into a single file.

For reference, the ABAC used in toolforge right now looks like this for a toolforge user account:

{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "pods", "apiGroup": "*"}, "kind": "Policy"}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "replicationcontrollers", "apiGroup": "*"}, "kind": "Policy"}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "services", "apiGroup": "*"}, "kind": "Policy"}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "secrets", "apiGroup": "*"}, "kind": "Policy"}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "deployments", "apiGroup": "*"}, "kind": "Policy"}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "replicasets", "apiGroup": "*"}, "kind": "Policy"}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "configmaps", "apiGroup": "*"}, "kind": "Policy"}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "jobs", "apiGroup": "*"}, "kind": "Policy"}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "cronjobs", "apiGroup": "*"}, "kind": "Policy"}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "spec": {"user": "mytool", "namespace": "mytool", "resource": "scheduledjobs", "apiGroup": "*"}, "kind": "Policy"}

This functionally restricts a tool very little within the context of their own namespace. The admission controllers do more to limit behavior. That should be relatively simple to combine into a single role for the namespace or similar along with a binding to the PSP that replaces the admission controllers.

Details

Event Timeline

Helpful person who also couldn't figure this out documented how they made it work on their blog: https://pmcgrath.net/using-pod-security-policies-with-kubeadm
Basically, we have to create RBACs and PSPs for the control plane pods so they will be created during init. I'll keep looking around and we can try different approaches to this. I tend to imagine that the api-server pods can be restarted with new settings by changing the right config maps. However, we've done a good job at making clusters fully reproducible/rebuildable so far. I tend to think that the right rbac policy files might be just the thing.

Good reference as well. I may start by trying this on my local minikube https://github.com/kubernetes/kubeadm/issues/791

This has more depth (noting for my reference--I don't expect blog type posts to work 100%, but the context is good). https://octetz.com/posts/setting-up-psps
I'll see if I can cook up a patch that will allow us to start playing with them after trying some things on minikube first.

Alright, experimenting a bit, I'm able to use kubeadm deploy with pod security policy. I'll tweak the policies where appropriate and see about adding them to puppet.

This basically just will add a step after kubeadm init and before the calico CRD apply with a policy that allows a node to deploy pods to the kube-system namespace with privileged mode. It will take some experimentation from there to ensure we have the right level of restrictions for toolforge users. It looks ultimately like we'll have a user in kubernetes per tool account, sort of like how we have it now except using roles. Those tool users will have to share certain role bindings to ensure they have the basic requirements of the rest of T215678 (since we believe this will account for 3 out of the four of those).

Change 524038 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: Enable pod security policy

https://gerrit.wikimedia.org/r/524038

Ok, so that's the basic cluster build and enablement portion. Now we need to sort out any additional RBAC and PSPs for the rest of the pods.

A blanket set of policies that applies to every authenticated user and implement the restrictions we want is probably good to cover most cases. We will want to also add a PSP that allows certain things to behave differently (possibly). Administrative tooling, maybe the admission webhook, etc.

Change 524038 merged by Bstorm:
[operations/puppet@production] toolforge: Enable pod security policy

https://gerrit.wikimedia.org/r/524038

When putting this out there, it was broken until I concatenated the whole file to the kubeadm-init.yaml....however some parts are not applied that way:

 kubeadm init --upload-certs --config /etc/kubernetes/kubeadm-init.yaml && cp /etc/kubernetes/admin.conf .kube/config
W0718 15:27:05.489071   24537 strict.go:47] unknown configuration schema.GroupVersionKind{Group:"policy", Version:"v1beta1", Kind:"PodSecurityPolicy"} for scheme definitions in "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go:31" and "k8s.io/kubernetes/cmd/kubeadm/app/componentconfigs/scheme.go:28"
[config] WARNING: Ignored YAML document with GroupVersionKind policy/v1beta1, Kind=PodSecurityPolicy
W0718 15:27:05.489155   24537 strict.go:47] unknown configuration schema.GroupVersionKind{Group:"rbac.authorization.k8s.io", Version:"v1", Kind:"ClusterRole"} for scheme definitions in "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go:31" and "k8s.io/kubernetes/cmd/kubeadm/app/componentconfigs/scheme.go:28"
[config] WARNING: Ignored YAML document with GroupVersionKind rbac.authorization.k8s.io/v1, Kind=ClusterRole
W0718 15:27:05.489179   24537 strict.go:47] unknown configuration schema.GroupVersionKind{Group:"rbac.authorization.k8s.io", Version:"v1", Kind:"RoleBinding"} for scheme definitions in "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go:31" and "k8s.io/kubernetes/cmd/kubeadm/app/componentconfigs/scheme.go:28"
[config] WARNING: Ignored YAML document with GroupVersionKind rbac.authorization.k8s.io/v1, Kind=RoleBinding

Change 524299 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: Enable pod security policy

https://gerrit.wikimedia.org/r/524299

Change 524299 merged by Bstorm:
[operations/puppet@production] toolforge: Enable pod security policy

https://gerrit.wikimedia.org/r/524299

Ok, the toolsbeta-test-k8s cluster now has PSP enabled and was built from scratch that way. Updated the build procedure here: T215531

Moving on to designing the PSPs and RBAC mechanism that manages it.

Bstorm added a comment.EditedJul 19 2019, 8:59 PM

So the most "free out of the box" authentication method presented in the current version of kubernetes as deployed with kubeadm is x.509 certs. The method of auth is enabled out of the box and includes an api for automating the process. Service accounts are not intended for user auth, and most other methods don't fit into existing workflows well in Toolforge. The process is ultimately just an automation that replaces maintain-kubeusers. Ideally, this will be something that runs inside kubernetes for redundancy. User groups can be included on the cert and easliy used for RBAC and therefore PSP.

In the future, actual user objects are planned for Kubernetes, but since they don't exist yet, this seems to be a simple method that is well-supported in a cluster such as we have deployed already (and was already planned or Toolforge or some time). An LDAP proxy would require a separate sign-in process, and this way we will be using serviceaccount tokens in a way that will be "future proof" by doing things the way the developers intend rather than trying to co-opt them or user auth.

As I write up the base proposal for RBAC/PSP, I'll figure that groups will be written to the certs from the information in LDAP according to toolforge needs (only include "tools.<name>" groups most likely)--then we can restrict or allow individuals to interact outside of tools as we wish. Tools themselves likely should have service accounts, but that's a slightly different matter--they should likely have regular accounts as well (which will only have one group).

I mention this here because it ties this process into what we'll be doing with T144153 and T228499

Change 524759 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: k8s: add PSP for nginx-ingress

https://gerrit.wikimedia.org/r/524759

Change 524803 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: put up a default psp to unblock other work

https://gerrit.wikimedia.org/r/524803

Change 524803 merged by Bstorm:
[operations/puppet@production] toolforge: put up a default psp to unblock other work

https://gerrit.wikimedia.org/r/524803

Change 524809 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] tooforge: actually place the default-psp file on the master server

https://gerrit.wikimedia.org/r/524809

Change 524809 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] tooforge: actually place the default-psp file on the master server

https://gerrit.wikimedia.org/r/524809

Bstorm updated the task description. (Show Details)Jul 29 2019, 5:42 PM

Change 524759 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: k8s: add nginx-ingress configuration.

https://gerrit.wikimedia.org/r/524759

Ok for Toolforge users, I currently have maintain-kubeusers generating individual role-bindings to the default "edit" clusterrole for new users in their namespace (T228499: Toolforge: changes to maintain-kubeusers). Since "edit" is a blank check of read/write access while preventing changes to RBAC/PSP, I thought it called for some modification. The biggest things I think we should remove from "edit" is:

  • the ability to interact with serviceaccounts (including impersonating them)
  • the modification of persistentvolumeclaims (since those will mostly be managed automatically and are important to toolforge functionality--aka NFS)
  • adding or changing poddisruptionbudgets (because it will interfere with admin ability to drain and manage pods)
  • manipulation of daemonsets because that's a terrible idea to allow toolforge users to do (effectively taking over cluster resources on, potentially, every single node)
  • changing autoscaling (seems like a mess).

Things I am intentionally leaving in:

  • networkpolicy -- seems potentially required for ingress, but that requires some more testing/research to be sure
  • ingress -- seems necessary to create ingress objects for each tool
  • secrets -- webservice or similar tooling is likely to use this one day or another
  • configmaps -- this is similar to secrets except that there is some concern that we might need to restrict at least one configmap (see T228499#5416711)

1# RBAC minimum perms for toolforge users, based on the "edit" system clusterrole with some permissions commented out:
2# verbs for R/O
3# ["get", "list", "watch"]
4
5# verbs for R/W
6# ["get", "list", "watch", "create", "update", "patch", "delete"]
7
8# resources needed by Toolforge users for sure:
9# pods, deployments,
10
11rules:
12- apiGroups:
13 - ""
14 resources:
15 - pods/attach
16 - pods/exec
17 - pods/portforward
18 - pods/proxy
19 - secrets
20 - services/proxy
21 verbs:
22 - get
23 - list
24 - watch
25# - apiGroups:
26# - ""
27# resources:
28# - serviceaccounts
29# verbs:
30# - impersonate
31- apiGroups:
32 - ""
33 resources:
34 - pods
35 - pods/attach
36 - pods/exec
37 - pods/portforward
38 - pods/proxy
39 verbs:
40 - create
41 - delete
42 - deletecollection
43 - patch
44 - update
45- apiGroups:
46 - ""
47 resources:
48 - configmaps
49 - endpoints
50# - persistentvolumeclaims
51 - replicationcontrollers
52 - replicationcontrollers/scale
53 - secrets
54# - serviceaccounts
55 - services
56 - services/proxy
57 verbs:
58 - create
59 - delete
60 - deletecollection
61 - patch
62 - update
63- apiGroups:
64 - apps
65 resources:
66# - daemonsets
67 - deployments
68 - deployments/rollback
69 - deployments/scale
70 - replicasets
71 - replicasets/scale
72 - statefulsets
73 - statefulsets/scale
74 verbs:
75 - create
76 - delete
77 - deletecollection
78 - patch
79 - update
80# - apiGroups:
81# - autoscaling
82# resources:
83# - horizontalpodautoscalers
84# verbs:
85# - create
86# - delete
87# - deletecollection
88# - patch
89# - update
90- apiGroups:
91 - batch
92 resources:
93 - cronjobs
94 - jobs
95 verbs:
96 - create
97 - delete
98 - deletecollection
99 - patch
100 - update
101- apiGroups:
102 - extensions
103 resources:
104# - daemonsets
105 - deployments
106 - deployments/rollback
107 - deployments/scale
108 - ingresses
109 - networkpolicies
110 - replicasets
111 - replicasets/scale
112 - replicationcontrollers/scale
113 verbs:
114 - create
115 - delete
116 - deletecollection
117 - patch
118 - update
119# - apiGroups:
120# - policy
121# resources:
122# - poddisruptionbudgets
123# verbs:
124# - create
125# - delete
126# - deletecollection
127# - patch
128# - update
129- apiGroups:
130 - networking.k8s.io
131 resources:
132 - ingresses
133 - networkpolicies
134 verbs:
135 - create
136 - delete
137 - deletecollection
138 - patch
139 - update
140- apiGroups:
141 - ""
142 resources:
143 - configmaps
144 - endpoints
145 - persistentvolumeclaims
146 - pods
147 - replicationcontrollers
148 - replicationcontrollers/scale
149# - serviceaccounts
150 - services
151 verbs:
152 - get
153 - list
154 - watch
155- apiGroups:
156 - ""
157 resources:
158 - bindings
159 - events
160 - limitranges
161 - namespaces/status
162 - pods/log
163 - pods/status
164 - replicationcontrollers/status
165 - resourcequotas
166 - resourcequotas/status
167 verbs:
168 - get
169 - list
170 - watch
171- apiGroups:
172 - ""
173 resources:
174 - namespaces
175 verbs:
176 - get
177 - list
178 - watch
179- apiGroups:
180 - apps
181 resources:
182 - controllerrevisions
183 - daemonsets
184 - deployments
185 - deployments/scale
186 - replicasets
187 - replicasets/scale
188 - statefulsets
189 - statefulsets/scale
190 verbs:
191 - get
192 - list
193 - watch
194- apiGroups:
195 - autoscaling
196 resources:
197 - horizontalpodautoscalers
198 verbs:
199 - get
200 - list
201 - watch
202- apiGroups:
203 - batch
204 resources:
205 - cronjobs
206 - jobs
207 verbs:
208 - get
209 - list
210 - watch
211- apiGroups:
212 - extensions
213 resources:
214 - daemonsets
215 - deployments
216 - deployments/scale
217 - ingresses
218 - networkpolicies
219 - replicasets
220 - replicasets/scale
221 - replicationcontrollers/scale
222 verbs:
223 - get
224 - list
225 - watch
226- apiGroups:
227 - policy
228 resources:
229 - poddisruptionbudgets
230 verbs:
231 - get
232 - list
233 - watch
234- apiGroups:
235 - networking.k8s.io
236 resources:
237 - ingresses
238 - networkpolicies
239 verbs:
240 - get
241 - list
242 - watch

Change 537732 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-kubernetes: restructure pod security policies

https://gerrit.wikimedia.org/r/537732

I've adapted my current test environment to using the PSPs in that patch and the proposed role above. So far, it behaves exactly as intended. A user with these credentials is nicely blind of the goings-on in any other namespace, but enjoys relative freedom to act within their own.

The PSP above covers preventing hostmounts other than what is intended.

I'll do a writeup describing the permissions in a somewhat clearer fashion than what we have above as well as the PSP as it currently stands. K8s documentation is sort of weak outside of API references, so this sort of experimentation seems to be the ideal path of design, unfortunately.

Change 537755 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: proposed role for all tools

https://gerrit.wikimedia.org/r/537755

aborrero assigned this task to Bstorm.Sep 25 2019, 3:28 PM
aborrero moved this task from Inbox to Important on the cloud-services-team (Kanban) board.
aborrero moved this task from Important to Doing on the cloud-services-team (Kanban) board.

Reallocating task.

Change 537732 merged by Bstorm:
[operations/puppet@production] toolforge-kubernetes: restructure pod security policies

https://gerrit.wikimedia.org/r/537732

Change 537755 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: proposed role for all tools

https://gerrit.wikimedia.org/r/537755

Bstorm closed this task as Resolved.Oct 31 2019, 10:19 PM

Moved the doc into the tree with the others. I think this is done for now.

Change 549108 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: Distribute the roles for toolforge users

https://gerrit.wikimedia.org/r/549108

Change 549108 merged by Bstorm:
[operations/puppet@production] toolforge: Distribute the roles for toolforge users

https://gerrit.wikimedia.org/r/549108

Change 549921 had a related patch set uploaded (by Phamhi; owner: Hieu Pham):
[operations/puppet@production] toolforge: Rename to toolforge-tool-role.yaml due to typo

https://gerrit.wikimedia.org/r/549921

Change 549921 merged by Phamhi:
[operations/puppet@production] toolforge: Rename to toolforge-tool-role.yaml due to typo

https://gerrit.wikimedia.org/r/549921