Page MenuHomePhabricator

Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection
Open, Needs TriagePublic

Description

In the Cloud VPS OpenStack environment we have an account named "novaobserver" that is associated with a role named "observer" which we add to every Cloud VPS project (T150092). The custom "observer" role is defined in our OpenStack configuration to allow it to inspect, but not change, a wide variety of OpenStack configuration related to each Cloud VPS project. The most visible use of this shared account is https://tools.wmflabs.org/openstack-browser/ which uses it extensively to interrogate the state of the OpenStack deployment.

We would like tools to be able to do similar inspection of cluster and namespace state in the Toolforge Kubernetes cluster. T201892: Toolforge: Build dashboard that breaks down webservice type usage by tool is one concrete example of a desired tool. Kubernetes equivalents of https://tools.wmflabs.org/sge-jobs/ and https://tools.wmflabs.org/sge-status/ would be other desired tools.

The "k8sobserver" role/user/whatever should NOT be able to see "private" or "secret" things in a namespace. This certainly includes Secret objects, and also probably should extend to ConfigMap objects. The approach taken with the novaobserver account is an allowlist rather than a blocklist so that we do not accidentally expose new things before we have made a reasoned examination of their security/privacy implications.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2019, 11:45 PM
Bstorm added a subscriber: Bstorm.Sep 19 2019, 11:54 PM

Oddly, since most serviceaccounts are namespaced, this may be easier to do with a simple "user" object with an x509 and a custom role that gives just the perms you need. We'll think more about it...

The "k8sobserver" role/user/whatever should NOT be able to see "private" or "secret" things in a namespace. This certainly includes Secret objects, and also probably should extend to ConfigMap objects.

I wonder how many ConfigMaps we actually have, depending on that maybe we could review them to check whether there's anything secret stored in there. We could have maintainers move secrets to Secret objects and then open up ConfigMap access.

Krenair added a comment.EditedSep 22 2019, 12:08 AM

The "k8sobserver" role/user/whatever should NOT be able to see "private" or "secret" things in a namespace. This certainly includes Secret objects, and also probably should extend to ConfigMap objects.

I wonder how many ConfigMaps we actually have, depending on that maybe we could review them to check whether there's anything secret stored in there. We could have maintainers move secrets to Secret objects and then open up ConfigMap access.

So I'm entirely new to this k8s cluster but, based on kubectl get --all-namespaces cm on tools-k8s-master-01, it looks like the answer is 2, and it looks like their contents would be fine to expose.

Bstorm added a comment.Oct 4 2019, 8:38 PM

On the new cluster, config maps are likely to serve additional purposes. Nobody SHOULD use configmaps for private or secret data. That doesn't guarantee they won't, but they shouldn't. There will be a configmap for each user at a minimum that will include the expiration date of their user certs (and is part of the instrumentation of their account).

The existing cluster on tools-k8s-master-01 is only passingly similar to the upgraded one in general. Tools that use the webservice command will get much of what they get now, with the addition of a bunch of other things.

If nothing else, perhaps this should clarify that it is for the new cluster. On the current cluster, I'd just make an admin user with a static token with a hand crafted ABAC thing.

That said, whether this is a service account in every namespace that the main account assumed the identity of or just a user with "cluster read", there's a default clusterrole called view in any recent version of k8s with rbac enabled that looks like this:

$ kubectl get clusterrole view -o yaml
aggregationRule:
  clusterRoleSelectors:
  - matchLabels:
      rbac.authorization.k8s.io/aggregate-to-view: "true"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2019-09-18T18:22:25Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
  name: view
  resourceVersion: "346"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/view
  uid: 6c101b9c-e546-4b70-a3af-e4168180d497
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - persistentvolumeclaims
  - pods
  - replicationcontrollers
  - replicationcontrollers/scale
  - serviceaccounts
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - bindings
  - events
  - limitranges
  - namespaces/status
  - pods/log
  - pods/status
  - replicationcontrollers/status
  - resourcequotas
  - resourcequotas/status
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - controllerrevisions
  - daemonsets
  - deployments
  - deployments/scale
  - replicasets
  - replicasets/scale
  - statefulsets
  - statefulsets/scale
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - deployments/scale
  - ingresses
  - networkpolicies
  - replicasets
  - replicasets/scale
  - replicationcontrollers/scale
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  - networkpolicies
  verbs:
  - get
  - list
  - watch

Basically, it sees all, and that's it. The only secret it would be able to view is its own credentialing one if it is using service accounts. If it is connected via a clusterrolebinding, then it can see all in all namespaces as well.

Bstorm added a comment.Oct 4 2019, 8:39 PM

Heh, in fact on the current cluster, while the user is easy, deploying anything is hard because there are no exceptions to the admission controllers. The new one will be more flexible.