Oddly, since most serviceaccounts are namespaced, this may be easier to do with a simple "user" object with an x509 and a custom role that gives just the perms you need. We'll think more about it...

The "k8sobserver" role/user/whatever should NOT be able to see "private" or "secret" things in a namespace. This certainly includes Secret objects, and also probably should extend to ConfigMap objects.

I wonder how many ConfigMaps we actually have, depending on that maybe we could review them to check whether there's anything secret stored in there. We could have maintainers move secrets to Secret objects and then open up ConfigMap access.

In T233372#5513239, @Krenair wrote:

The "k8sobserver" role/user/whatever should NOT be able to see "private" or "secret" things in a namespace. This certainly includes Secret objects, and also probably should extend to ConfigMap objects.

I wonder how many ConfigMaps we actually have, depending on that maybe we could review them to check whether there's anything secret stored in there. We could have maintainers move secrets to Secret objects and then open up ConfigMap access.

So I'm entirely new to this k8s cluster but, based on kubectl get --all-namespaces cm on tools-k8s-master-01, it looks like the answer is 2, and it looks like their contents would be fine to expose.

On the new cluster, config maps are likely to serve additional purposes. Nobody SHOULD use configmaps for private or secret data. That doesn't guarantee they won't, but they shouldn't. There will be a configmap for each user at a minimum that will include the expiration date of their user certs (and is part of the instrumentation of their account).

The existing cluster on tools-k8s-master-01 is only passingly similar to the upgraded one in general. Tools that use the webservice command will get much of what they get now, with the addition of a bunch of other things.

If nothing else, perhaps this should clarify that it is for the new cluster. On the current cluster, I'd just make an admin user with a static token with a hand crafted ABAC thing.

That said, whether this is a service account in every namespace that the main account assumed the identity of or just a user with "cluster read", there's a default clusterrole called view in any recent version of k8s with rbac enabled that looks like this:

$ kubectl get clusterrole view -o yaml
aggregationRule:
  clusterRoleSelectors:
  - matchLabels:
      rbac.authorization.k8s.io/aggregate-to-view: "true"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2019-09-18T18:22:25Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
  name: view
  resourceVersion: "346"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/view
  uid: 6c101b9c-e546-4b70-a3af-e4168180d497
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - persistentvolumeclaims
  - pods
  - replicationcontrollers
  - replicationcontrollers/scale
  - serviceaccounts
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - bindings
  - events
  - limitranges
  - namespaces/status
  - pods/log
  - pods/status
  - replicationcontrollers/status
  - resourcequotas
  - resourcequotas/status
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - controllerrevisions
  - daemonsets
  - deployments
  - deployments/scale
  - replicasets
  - replicasets/scale
  - statefulsets
  - statefulsets/scale
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - deployments/scale
  - ingresses
  - networkpolicies
  - replicasets
  - replicasets/scale
  - replicationcontrollers/scale
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  - networkpolicies
  verbs:
  - get
  - list
  - watch

Basically, it sees all, and that's it. The only secret it would be able to view is its own credentialing one if it is using service accounts. If it is connected via a clusterrolebinding, then it can see all in all namespaces as well.

Heh, in fact on the current cluster, while the user is easy, deploying anything is hard because there are no exceptions to the admission controllers. The new one will be more flexible.

On the new cluster, config maps are likely to serve additional purposes. Nobody SHOULD use configmaps for private or secret data. That doesn't guarantee they won't, but they shouldn't. There will be a configmap for each user at a minimum that will include the expiration date of their user certs (and is part of the instrumentation of their account).

The existing cluster on tools-k8s-master-01 is only passingly similar to the upgraded one in general. Tools that use the webservice command will get much of what they get now, but they will need a bit more in every case...plus the new cluster will have RBAC at all.

I am also hoping to do fewer host mounts and more config maps for most of the rest of the data exposed to pods (later edit after coming back to this: nope not right now, lol)

Overall, I do think that the default 'view' clusterrole is suitable, partly because we are likely to want to be able to track new services. Just need a quick script or doc to enable an appropriate service account with that role in a tool namespace on request.

@bd808 Do you have a tool you'd like to include in this initially? That would make it easier for me, honestly, to reason through a script or process to enable it.

Writing up a script that can be dropped on the control plane nodes that will allow any arbitrary tool to be granted a service account that has access to the view role on request.

In T233372#5752136, @Bstorm wrote:

@bd808 Do you have a tool you'd like to include in this initially? That would make it easier for me, honestly, to reason through a script or process to enable it.

The k8s-status tool would be a good candidate. That's where I would like to build some kind of "what is active on the Toolforge k8s cluster" website.

Almost done with a script to test. Just need to set up a read-only PSP that we can grant it across the cluster as well as RBAC.

It might be sufficient to use the tool's psp

Ok I have a script that works nicely. The only thing I kind of dislike is that it can see all pods and configmaps in kube-system. That's not really a problem, I suppose. None of that is secret (in fact, most is effectively in puppet). I'm sure some places would want to rope that off, though :)

Example below from a pod created with the serviceaccount specified, using any old image, I downloaded kubectl to the /tmp directory and ran it (kubectl automagically looks for the service account creds).

$ ls
kubectl
$ ./kubectl get pods --all-namespaces
NAMESPACE            NAME                                                   READY   STATUS    RESTARTS   AGE
ingress-admission    ingress-admission-55fb8554b5-j9xds                     1/1     Running   1          6d21h
ingress-admission    ingress-admission-55fb8554b5-wmr46                     1/1     Running   0          6d21h
ingress-nginx        nginx-ingress-64dc7c9c57-jkn75                         1/1     Running   1          6d2h
ingress-nginx        nginx-ingress-64dc7c9c57-kpkk4                         1/1     Running   0          6d2h
ingress-nginx        nginx-ingress-64dc7c9c57-nzm76                         1/1     Running   0          6d2h
kube-system          calico-kube-controllers-59f54d6bbc-dwg76               1/1     Running   1          15d
kube-system          calico-node-2sf9d                                      1/1     Running   5          56d
kube-system          calico-node-dfbqd                                      1/1     Running   1          56d
kube-system          calico-node-g4hr7                                      1/1     Running   2          56d
kube-system          calico-node-lp2c9                                      1/1     Running   0          7d22h

So any tool that I grant this to can use a particular service account to see into almost everything. It cannot run with any greater priv escalation than your usual account and actually has less read/write powers (basically none). Webservice doesn't allow injecting a service account yet (but this is a good future feature in a way), so a start/stop script that reads yaml sounds like the way to go.

Patch incoming...

Change 559212 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: add a script to grant "observer" access to a tool

https://gerrit.wikimedia.org/r/559212

In T233372#5752414, @Bstorm wrote:

Webservice doesn't allow injecting a service account yet (but this is a good future feature in a way), so a start/stop script that reads yaml sounds like the way to go.

I imagine the actual tool I would build would be a flask app using the k8s python client library, so I don't think that webservice actually needs anything fancy? Or is it somehow not possible to call back to the k8s API server using different credentials than the ones that the pod running the code is using?

It does. Unless I change that script to grant the "default" service account (that's it's actual name) for your namespace these privileges, then you need to specify a service account on launch of the pod or more likely deployment managing the pod.

I can provide a template or script for that, and it'll make perfect sense when you see it, if needed. Everything has a service account, and not specifying one gives you the "default" one, which maintain-kubeusers gives the rights of a tools user (so that webservice-created replicasets get the PSP needed to start pods otherwise, that service account can only do basic pod ops). It's considered sort of a bad practice to use the default SA in a pod since the SA defines the perms of a service within k8s (like how an x509 defines a human or outside user's perms), but it is "ok", so I left that alone in webservice and worked around it because it would be a pain to introduce to all users. That's part of that whole, "don't force your devs to directly interact with kubernetes a lot" notion. I figure we can adapt our tooling more to that idea once we are entirely on a cluster that supports all the APIs at least.

So yeah, I should say that I *can* just give the default service account of the namespace full cluster read-only (allowing webservice to work as usual), but I don't *want* to. I'd rather just provide a start/stop script that sets the SA in the few tools that need this, if it's all the same to you. That also makes it easier to remove the perms if said tool stops needing it.

Here's a nice script you could use to launch...you'll notice two things different here than what you get from webservice. For one, the deployment is in api group apps/v1, that means you don't have to do all that label-searching because cascading deletes work (thus my frustration with pykube). Second, the service account is set in the template for the deployment. That and it merely suggests what to do to restart the app instead of doing what webservice does to take some strain off the ingress controllers.

It sets the toolforge: tool label because that's what mounts your NFS dirs, SSSD stuff and sets your home env var.

The basic idea is that this clearly introduces the need for a serviceaccount field in webservice, unless we change tooling a lot.

Change 559212 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: add a script to grant "observer" access to a tool

https://gerrit.wikimedia.org/r/559212

If this process works for you for now (we can probably hack in a service-account setting for webservice if we ever want to...but the API versions I used in the bash script are actually likely to work slightly better because they aren't deprecated), I can run the script to give the service account to the tool namespace any time.

Added https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Maintenance#wmcs-k8s-enable-cluster-monitor for documenting the process.

@bd808
You should be clear to start fiddling with k8s-status on the new cluster when you migrate it now:

root@tools-k8s-control-1:~# wmcs-k8s-enable-cluster-monitor k8s-status
NAME              STATUS   AGE
tool-k8s-status   Active   2d17h
Creating the service account...
serviceaccount/k8s-status-obs created
Enabling read-only access to the cluster...
clusterrolebinding.rbac.authorization.k8s.io/k8s-status-obs created
rolebinding.rbac.authorization.k8s.io/k8s-status-obs-psp created
NAME             SECRETS   AGE
k8s-status-obs   1         1s
*********************
Done!

I'd just use P9946 or something similar for starting and stopping the service.