Page MenuHomePhabricator

Allow kubernetes observer role to read "nodes" data
Closed, ResolvedPublic

Description

Follow up to T233372: Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection. I would like the k8s-status tool to show the number of nodes in the cluster and possibly some data on how much CPU/RAM is in use.

>>> import kubernetes
>>> kubernetes.config.load_incluster_config()
>>> v1 = kubernetes.client.CoreV1Api()
>>> v1.list_node()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13267, in list_node
    (data) = self.list_node_with_http_info(**kwargs)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13361, in list_node_with_http_info
    collection_formats=collection_formats)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 355, in request
    headers=headers)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in GET
    query_params=query_params)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 23 Dec 2019 18:01:23 GMT', 'Content-Length': '292'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes is forbidden: User \"system:serviceaccount:tool-k8s-status:k8s-status-obs\" cannot list resource \"nodes\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"nodes"},"code":403}

Event Timeline

bd808 triaged this task as Medium priority.Dec 23 2019, 6:45 PM

I think this would be fixed by adding this snippet to clusterrole "view" or making a new clusterrole of our own including it and the existing "view" permissions:

- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch

I strongly favor using our own clusterrole to preserve expectations (and changes between upgrades/rebuilds). I should have known that wouldn't be available to a system default read-only clusterrole :)

The role should be added to the base psp/roles we create at spin-up, and then the script can be updated to use that clusterrole instead.

@bd808: I was constructing a custom clusterrole when I noticed that you *do* have node metrics in view thanks to aggregation, which is likely to get you what you want anyway.

It includes:

- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch

The only unfortunate thing about this is that the API isn't included in the kubernetes client so you have to do this: https://github.com/kubernetes-client/python/issues/474

That would give you the same info as kubectl top nodes and kubectl top pods commands.

This is similar to what I expect that to get you:

root@toolsbeta-test-k8s-control-1:~# kubectl top nodes
NAME                           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
toolsbeta-test-k8s-control-1   165m         8%     1525Mi          39%       
toolsbeta-test-k8s-control-2   214m         10%    1651Mi          42%       
toolsbeta-test-k8s-control-3   161m         8%     1582Mi          41%       
toolsbeta-test-k8s-worker-1    878m         43%    2437Mi          63%       
toolsbeta-test-k8s-worker-2    436m         21%    1340Mi          34%       
toolsbeta-test-k8s-worker-3    316m         7%     3992Mi          50%

Please lemme know if I am wrong.

There is quite a bit of data in the metrics.k8s.io APIs. I will work on adding them into the app and see if that gets us all the way to what we are wanting for visualization right now.

>>> import kubernetes
>>> kubernetes.config.load_incluster_config()
>>> cust = kubernetes.client.CustomObjectsApi()
>>> import pprint
>>> pprint.pprint(cust.list_cluster_custom_object('metrics.k8s.io', 'v1beta1', 'nodes'))
{'apiVersion': 'metrics.k8s.io/v1beta1',
 'items': [{'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-4',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-4'},
            'timestamp': '2020-01-02T20:15:41Z',
            'usage': {'cpu': '115889991n', 'memory': '2202572Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-5',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-5'},
            'timestamp': '2020-01-02T20:15:43Z',
            'usage': {'cpu': '435133527n', 'memory': '1670660Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-control-1',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-control-1'},
            'timestamp': '2020-01-02T20:15:46Z',
            'usage': {'cpu': '139717252n', 'memory': '2029708Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-control-2',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-control-2'},
            'timestamp': '2020-01-02T20:15:41Z',
            'usage': {'cpu': '109962548n', 'memory': '1695480Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-control-3',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-control-3'},
            'timestamp': '2020-01-02T20:15:42Z',
            'usage': {'cpu': '109308931n', 'memory': '1693756Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-1',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-1'},
            'timestamp': '2020-01-02T20:15:41Z',
            'usage': {'cpu': '131675178n', 'memory': '1576956Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-2',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-2'},
            'timestamp': '2020-01-02T20:15:46Z',
            'usage': {'cpu': '111128381n', 'memory': '1518384Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-3',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-3'},
            'timestamp': '2020-01-02T20:15:48Z',
            'usage': {'cpu': '186817490n', 'memory': '2050004Ki'},
            'window': '30s'}],
 'kind': 'NodeMetricsList',
 'metadata': {'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes'}}
>>> pprint.pprint(cust.list_cluster_custom_object('metrics.k8s.io', 'v1beta1', 'pods'))
{'apiVersion': 'metrics.k8s.io/v1beta1',
 'items': [{'containers': [{'name': 'coredns',
                            'usage': {'cpu': '3382319n', 'memory': '23148Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'coredns-5c98db65d4-hq6hc',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-5c98db65d4-hq6hc'},
            'timestamp': '2020-01-02T20:17:33Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '699798n', 'memory': '21768Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-wkgzb',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-wkgzb'},
            'timestamp': '2020-01-02T20:17:31Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '433690n', 'memory': '20208Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-77qj4',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-77qj4'},
            'timestamp': '2020-01-02T20:17:46Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '4230284n', 'memory': '20148Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-8sl46',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-8sl46'},
            'timestamp': '2020-01-02T20:17:38Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '25287162n',
                                      'memory': '219160Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-qz4tn',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-qz4tn'},
            'timestamp': '2020-01-02T20:17:40Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '387686n', 'memory': '17176Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-7ssq4',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-7ssq4'},
            'timestamp': '2020-01-02T20:17:44Z',
            'window': '30s'},
           {'containers': [{'name': 'webhook',
                            'usage': {'cpu': '18097n', 'memory': '4956Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'ingress-admission-55fb8554b5-btp7h',
                         'namespace': 'ingress-admission',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-admission/pods/ingress-admission-55fb8554b5-btp7h'},
            'timestamp': '2020-01-02T20:17:42Z',
            'window': '30s'},
           {'containers': [{'name': 'webhook',
                            'usage': {'cpu': '27464n', 'memory': '5744Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'registry-admission-6f5f6589c5-tzzk5',
                         'namespace': 'registry-admission',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/registry-admission/pods/registry-admission-6f5f6589c5-tzzk5'},
            'timestamp': '2020-01-02T20:17:47Z',
            'window': '30s'},
           {'containers': [{'name': 'nginx-ingress-controller',
                            'usage': {'cpu': '3586290n',
                                      'memory': '171204Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'nginx-ingress-64dc7c9c57-2j7qd',
                         'namespace': 'ingress-nginx',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-nginx/pods/nginx-ingress-64dc7c9c57-2j7qd'},
            'timestamp': '2020-01-02T20:17:39Z',
            'window': '30s'},
           {'containers': [{'name': 'nginx-ingress-controller',
                            'usage': {'cpu': '12302417n',
                                      'memory': '166668Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'nginx-ingress-64dc7c9c57-hsxkj',
                         'namespace': 'ingress-nginx',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-nginx/pods/nginx-ingress-64dc7c9c57-hsxkj'},
            'timestamp': '2020-01-02T20:17:41Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '66255n', 'memory': '207468Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'k8s-status-7858fc66cd-6t2tg',
                         'namespace': 'tool-k8s-status',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-k8s-status/pods/k8s-status-7858fc66cd-6t2tg'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'maintain-kubeusers',
                            'usage': {'cpu': '76121n', 'memory': '23904Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'maintain-kubeusers-ops',
                         'namespace': 'maintain-kubeusers',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/maintain-kubeusers/pods/maintain-kubeusers-ops'},
            'timestamp': '2020-01-02T20:17:46Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '147589n', 'memory': '36468Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'bd808-test-6b88576f69-qttbn',
                         'namespace': 'tool-bd808-test',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-bd808-test/pods/bd808-test-6b88576f69-qttbn'},
            'timestamp': '2020-01-02T20:17:47Z',
            'window': '30s'},
           {'containers': [{'name': 'webhook',
                            'usage': {'cpu': '13916n', 'memory': '4924Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'ingress-admission-55fb8554b5-6c98v',
                         'namespace': 'ingress-admission',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-admission/pods/ingress-admission-55fb8554b5-6c98v'},
            'timestamp': '2020-01-02T20:17:40Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-apiserver',
                            'usage': {'cpu': '18771837n',
                                      'memory': '365324Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-apiserver-tools-k8s-control-3',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-apiserver-tools-k8s-control-3'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '18705688n',
                                      'memory': '220284Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-44ntg',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-44ntg'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'bot',
                            'usage': {'cpu': '966472n', 'memory': '58592Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'stashbot.bot-5bbf56f756-pgxmn',
                         'namespace': 'tool-stashbot',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-stashbot/pods/stashbot.bot-5bbf56f756-pgxmn'},
            'timestamp': '2020-01-02T20:17:48Z',
            'window': '30s'},
           {'containers': [{'name': 'interactive',
                            'usage': {'cpu': '0', 'memory': '48956Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'interactive',
                         'namespace': 'tool-k8s-status',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-k8s-status/pods/interactive'},
            'timestamp': '2020-01-02T20:17:37Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-scheduler',
                            'usage': {'cpu': '765u', 'memory': '12500Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-scheduler-tools-k8s-control-3',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-tools-k8s-control-3'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'metrics-server',
                            'usage': {'cpu': '738180n', 'memory': '23584Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'metrics-server-575d5f6d95-ff7pp',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/metrics-server-575d5f6d95-ff7pp'},
            'timestamp': '2020-01-02T20:17:39Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-controller-manager',
                            'usage': {'cpu': '407090n', 'memory': '13308Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-controller-manager-tools-k8s-control-3',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-controller-manager-tools-k8s-control-3'},
            'timestamp': '2020-01-02T20:17:46Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '4566727n', 'memory': '19984Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-qvfvp',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-qvfvp'},
            'timestamp': '2020-01-02T20:17:43Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-kube-controllers',
                            'usage': {'cpu': '1380455n', 'memory': '18504Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-kube-controllers-59f54d6bbc-mwrjm',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-kube-controllers-59f54d6bbc-mwrjm'},
            'timestamp': '2020-01-02T20:17:36Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '16508472n',
                                      'memory': '204652Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-c8rxg',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-c8rxg'},
            'timestamp': '2020-01-02T20:17:48Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-scheduler',
                            'usage': {'cpu': '562934n', 'memory': '13180Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-scheduler-tools-k8s-control-2',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-tools-k8s-control-2'},
            'timestamp': '2020-01-02T20:17:34Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '31960738n',
                                      'memory': '215788Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-snn29',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-snn29'},
            'timestamp': '2020-01-02T20:17:32Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '24848227n',
                                      'memory': '213428Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-nrk46',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-nrk46'},
            'timestamp': '2020-01-02T20:17:44Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-controller-manager',
                            'usage': {'cpu': '353369n', 'memory': '12744Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-controller-manager-tools-k8s-control-2',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-controller-manager-tools-k8s-control-2'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '354084n', 'memory': '17128Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-8fv9z',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-8fv9z'},
            'timestamp': '2020-01-02T20:17:46Z',
            'window': '30s'},
           {'containers': [{'name': 'maintain-kubeusers',
                            'usage': {'cpu': '350151364n',
                                      'memory': '63260Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'maintain-kubeusers-6d9c45f4bc-5bqq5',
                         'namespace': 'maintain-kubeusers',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/maintain-kubeusers/pods/maintain-kubeusers-6d9c45f4bc-5bqq5'},
            'timestamp': '2020-01-02T20:17:47Z',
            'window': '30s'},
           {'containers': [{'name': 'nginx-ingress-controller',
                            'usage': {'cpu': '3229161n',
                                      'memory': '202184Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'nginx-ingress-64dc7c9c57-twsvk',
                         'namespace': 'ingress-nginx',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-nginx/pods/nginx-ingress-64dc7c9c57-twsvk'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-apiserver',
                            'usage': {'cpu': '152427205n',
                                      'memory': '299496Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-apiserver-tools-k8s-control-2',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-apiserver-tools-k8s-control-2'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '163393n', 'memory': '35128Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'replag-d9b46cb69-gjdkq',
                         'namespace': 'tool-replag',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-replag/pods/replag-d9b46cb69-gjdkq'},
            'timestamp': '2020-01-02T20:17:47Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-state-metrics',
                            'usage': {'cpu': '1603192n', 'memory': '40536Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-state-metrics-7678f4fc5c-59627',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-state-metrics-7678f4fc5c-59627'},
            'timestamp': '2020-01-02T20:17:34Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '18829429n',
                                      'memory': '185148Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-g64gn',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-g64gn'},
            'timestamp': '2020-01-02T20:17:36Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '21814334n',
                                      'memory': '213800Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-pcmlx',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-pcmlx'},
            'timestamp': '2020-01-02T20:17:43Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-apiserver',
                            'usage': {'cpu': '29630223n',
                                      'memory': '353296Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-apiserver-tools-k8s-control-1',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-apiserver-tools-k8s-control-1'},
            'timestamp': '2020-01-02T20:17:48Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-scheduler',
                            'usage': {'cpu': '1511734n', 'memory': '14828Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-scheduler-tools-k8s-control-1',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-tools-k8s-control-1'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '505140n', 'memory': '27020Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-wwmjc',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-wwmjc'},
            'timestamp': '2020-01-02T20:17:44Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-controller-manager',
                            'usage': {'cpu': '9742817n',
                                      'memory': '231576Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-controller-manager-tools-k8s-control-1',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-controller-manager-tools-k8s-control-1'},
            'timestamp': '2020-01-02T20:17:41Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '412237n', 'memory': '22764Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-r94mq',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-r94mq'},
            'timestamp': '2020-01-02T20:17:41Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '18771922n',
                                      'memory': '216352Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-dn2pk',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-dn2pk'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'coredns',
                            'usage': {'cpu': '3255301n', 'memory': '24632Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'coredns-5c98db65d4-97hjp',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-5c98db65d4-97hjp'},
            'timestamp': '2020-01-02T20:17:35Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '132148n', 'memory': '29368Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'stashbot-85cc8c6c55-lz54v',
                         'namespace': 'tool-stashbot',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-stashbot/pods/stashbot-85cc8c6c55-lz54v'},
            'timestamp': '2020-01-02T20:17:42Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '53184n', 'memory': '45156Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'bash-666c74f75f-4zsc6',
                         'namespace': 'tool-bash',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-bash/pods/bash-666c74f75f-4zsc6'},
            'timestamp': '2020-01-02T20:17:39Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '76570007n',
                                      'memory': '205900Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'fourohfour-787f7f88f9-npgx5',
                         'namespace': 'tool-fourohfour',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-fourohfour/pods/fourohfour-787f7f88f9-npgx5'},
            'timestamp': '2020-01-02T20:17:33Z',
            'window': '30s'},
           {'containers': [{'name': 'webhook',
                            'usage': {'cpu': '221956n', 'memory': '7632Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'registry-admission-6f5f6589c5-clj2w',
                         'namespace': 'registry-admission',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/registry-admission/pods/registry-admission-6f5f6589c5-clj2w'},
            'timestamp': '2020-01-02T20:17:40Z',
            'window': '30s'}],
 'kind': 'PodMetricsList',
 'metadata': {'selfLink': '/apis/metrics.k8s.io/v1beta1/pods'}}

The fact that it reports CPU in "nanoCPUs"... Apparently someone thinks that's how finely you need to slice this kind of thing.

I have added some data from the metrics api to https://tools.wmflabs.org/k8s-status/. This does allow us to show:

  • total number of control nodes
  • total number of worker nodes
  • ram in use
  • cpu in use

It does not let us show ram & cpu available which I think would be interesting. Exposing the node API would give access to a lot of metadata about each node, including the allocatable CPU and RAM:

$ kubectl get nodes tools-k8s-worker-4 -o yaml
apiVersion: v1
kind: Node
metadata:
  annotations:
    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
    node.alpha.kubernetes.io/ttl: "0"
    projectcalico.org/IPv4Address: 172.16.0.113/21
    projectcalico.org/IPv4IPIPTunnelAddr: 192.168.81.0
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2019-11-29T12:12:34Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: tools-k8s-worker-4
    kubernetes.io/os: linux
  name: tools-k8s-worker-4
  resourceVersion: "11111196"
  selfLink: /api/v1/nodes/tools-k8s-worker-4
  uid: e130408d-c677-4a53-94ac-ae58aa1976c5
spec:
  podCIDR: 192.168.6.0/24
status:
  addresses:
  - address: 172.16.0.113
    type: InternalIP
  - address: tools-k8s-worker-4
    type: Hostname
  allocatable:
    cpu: "4"
    ephemeral-storage: "17951053795"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 8067460Ki
    pods: "110"
  capacity:
    cpu: "4"
    ephemeral-storage: 19478140Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 8169860Ki
    pods: "110"
  conditions:
  - lastHeartbeatTime: "2019-12-12T12:19:19Z"
    lastTransitionTime: "2019-12-12T12:19:19Z"
    message: Calico is running on this node
    reason: CalicoIsUp
    status: "False"
    type: NetworkUnavailable
  - lastHeartbeatTime: "2020-01-03T05:40:15Z"
    lastTransitionTime: "2019-11-29T12:12:34Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2020-01-03T05:40:15Z"
    lastTransitionTime: "2019-11-29T12:12:34Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2020-01-03T05:40:15Z"
    lastTransitionTime: "2019-11-29T12:12:34Z"
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: "2020-01-03T05:40:15Z"
    lastTransitionTime: "2019-11-29T12:12:44Z"
    message: kubelet is posting ready status. AppArmor enabled
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-jdk11-sssd-web@sha256:0d397ff6c41a29426c059e5b3c2f07303c27d000c853719c94222869af9ab734
    - docker-registry.tools.wmflabs.org/toolforge-jdk11-sssd-web:latest
    sizeBytes: 1059181798
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web@sha256:ecf1b20d28eac60c415e6c8b815b9c3e140ee56b4a7462ca6a0936bc62142713
    sizeBytes: 919889321
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-golang-sssd-web@sha256:32cd42bc997ca1ba6e1e64accb5363bb0116ad9e55e4e2aa9ff4b23d25494633
    - docker-registry.tools.wmflabs.org/toolforge-golang-sssd-web:latest
    sizeBytes: 891721425
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base@sha256:25f796e556a62b20e712f23492df89102a2680933bada363ad7a8a974f0e944a
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base:latest
    sizeBytes: 886712537
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python35-sssd-web@sha256:5bd9dab68c52e11be793f146f27f22d63fb6c41fe084ad4a0734d3bc41287867
    - docker-registry.tools.wmflabs.org/toolforge-python35-sssd-web:latest
    sizeBytes: 836890582
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python35-sssd-web@sha256:5de157b4ec83c060eafd985cdb715242d0beed8f15a30d9fe29aeae617cca3c1
    sizeBytes: 836865747
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-golang111-sssd-web@sha256:f27a711eca47af1566c2ecdc663bac40010adea7b368a4be543534e4428fb7da
    - docker-registry.tools.wmflabs.org/toolforge-golang111-sssd-web:latest
    sizeBytes: 833649533
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-node10-sssd-web@sha256:c726accc1ff69d0aab0d6e7c953aa8a247dbf2adcbed4875237193ab4220c5cb
    - docker-registry.tools.wmflabs.org/toolforge-node10-sssd-web:latest
    sizeBytes: 784062125
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web@sha256:25bb8ecc642f4d52e4403b39c00fddb7a2abbaa9488c7ac0db8116f3cb57b0e9
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web:latest
    sizeBytes: 767916282
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-jdk8-sssd-web@sha256:114c594cf54584a2ce550511340efb6b4534a6774b3a74fb06f6d5c3912b3471
    - docker-registry.tools.wmflabs.org/toolforge-jdk8-sssd-web:latest
    sizeBytes: 739886916
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-ruby25-sssd-web@sha256:444f4fc416c589b027d340799b014d3f72a75b64ca6bd38145bb745495535efb
    - docker-registry.tools.wmflabs.org/toolforge-ruby25-sssd-web:latest
    sizeBytes: 736192302
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python34-sssd-web@sha256:24c92be4136258b1a0ad810184e0a8956e930cb9be0aee93fa2d34cc8953ff6a
    - docker-registry.tools.wmflabs.org/toolforge-python34-sssd-web:latest
    sizeBytes: 713162449
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python2-sssd-web@sha256:b3dbbd87d2b1c21491db568894278ab866cc390f9ecaa9768cbbc55628cc8ea9
    - docker-registry.tools.wmflabs.org/toolforge-python2-sssd-web:latest
    sizeBytes: 685182640
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-node6-sssd-web@sha256:49b2df0004e0b4bec1da29462b54a882c6dc15d315fa90b935966cb5b6437a5d
    - docker-registry.tools.wmflabs.org/toolforge-node6-sssd-web:latest
    sizeBytes: 614698538
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web@sha256:7454a32bf6ca360dd56b3e4513b6329cbc08a192faa1c015afab88ddbf12d77a
    - docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest
    sizeBytes: 591205517
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-ruby21-sssd-web@sha256:91695f4e72b00d16f2cb0105bc0507bcce428c86dbfe0be17f00cbaae0118785
    - docker-registry.tools.wmflabs.org/toolforge-ruby21-sssd-web:latest
    sizeBytes: 551482292
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-php72-sssd-web@sha256:7fa16aeaa23b44d550059944749d7df47558b46257d5bdd98d29da1d67604d5e
    - docker-registry.tools.wmflabs.org/toolforge-php72-sssd-web:latest
    sizeBytes: 538009244
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-php5-sssd-web@sha256:3d293a785228cdef5e0f34592737126a6f70b014f8c6ed8e57c4f6da58f56075
    - docker-registry.tools.wmflabs.org/toolforge-php5-sssd-web:latest
    sizeBytes: 531993968
  - names:
    - docker-registry.tools.wmflabs.org/nginx-ingress-controller@sha256:b9cd638b8849f25210740b075d27ef2e55ffd2861488ead98276aa70b8a859ab
    - docker-registry.tools.wmflabs.org/nginx-ingress-controller:0.25.1
    sizeBytes: 510687563
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-tcl86-sssd-web@sha256:a35c12693cc3991ccb646670a56108f64c12240a91f110d50618f6cd05fc5f5e
    - docker-registry.tools.wmflabs.org/toolforge-tcl86-sssd-web:latest
    sizeBytes: 469721313
  - names:
    - calico/node@sha256:6679ccc9f19dba3eb084db991c788dc9661ad3b5d5bafaa3379644229dca6b05
    - calico/node:v3.8.0
    sizeBytes: 154769666
  - names:
    - calico/cni@sha256:decba0501ab0658e6e7da2f5625f1eabb8aba5690f9206caba3bf98caca5094c
    - calico/cni:v3.8.0
    sizeBytes: 143333412
  - names:
    - k8s.gcr.io/kube-proxy@sha256:ef245ddefef697c8b42611c237603acf41bfdb2b7ec3e434b7c3592864dcfff8
    - k8s.gcr.io/kube-proxy:v1.15.6
    sizeBytes: 82412380
  - names:
    - quay.io/coreos/kube-state-metrics@sha256:f75c3e5c5c7f65846ddd6883d6187b38f77721a3938f241c9e5d0ebe7beb8e19
    - quay.io/coreos/kube-state-metrics:v1.8.0
    sizeBytes: 33616576
  - names:
    - calico/pod2daemon-flexvol@sha256:6ec8b823e5ce3440318edfcdd2ab8b6660110782713f24f53dac5a3c227afb11
    - calico/pod2daemon-flexvol:v3.8.0
    sizeBytes: 9371181
  - names:
    - docker-registry.tools.wmflabs.org/pause@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610
    - docker-registry.tools.wmflabs.org/pause:3.1
    sizeBytes: 742472
  nodeInfo:
    architecture: amd64
    bootID: 6c6134fa-b225-46d7-98f1-72132e34bce2
    containerRuntimeVersion: docker://19.3.5
    kernelVersion: 4.19.0-6-amd64
    kubeProxyVersion: v1.15.6
    kubeletVersion: v1.15.6
    machineID: 90c9b8c4c3ac49b38a17edd815457e2a
    operatingSystem: linux
    osImage: Debian GNU/Linux 10 (buster)
    systemUUID: 90c9b8c4-c3ac-49b3-8a17-edd815457e2a

I found me another reason I'd rather not use the "view" clusterrole anyway. Patch incoming :)

Change 561889 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: switch cluster monitor script to using a new clusterrole

https://gerrit.wikimedia.org/r/561889

Change 561889 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: switch cluster monitor script to using a new clusterrole

https://gerrit.wikimedia.org/r/561889

Change 561891 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: renamed files don't get changed

https://gerrit.wikimedia.org/r/561891

Change 561891 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: renamed files don't get changed

https://gerrit.wikimedia.org/r/561891

root@tools-k8s-control-1:~# kubectl delete clusterrolebindings k8s-status-obs
clusterrolebinding.rbac.authorization.k8s.io "k8s-status-obs" deleted
root@tools-k8s-control-1:~# wmcs-k8s-enable-cluster-monitor k8s-status
NAME              STATUS   AGE
tool-k8s-status   Active   17d
Creating the service account...
serviceaccount/k8s-status-obs unchanged
Enabling read-only access to the cluster...
clusterrolebinding.rbac.authorization.k8s.io/k8s-status-obs created
rolebinding.rbac.authorization.k8s.io/k8s-status-obs-psp unchanged
NAME             SECRETS   AGE
k8s-status-obs   1         14d
*********************
Done!

Ok, you should be using the new role now. Let me know how that works for you.

Notably, the tool is still working on the web, which is good 😁

bd808 assigned this task to Bstorm.

Works!

>>> import k8s.client
>>> v1 = k8s.client.corev1_client()
>>> v1.list_node().items[0]
{'api_version': None,
 'kind': None,
 'metadata': {'annotations': {'kubeadm.alpha.kubernetes.io/cri-socket': '/var/run/dockershim.sock',
                              'node.alpha.kubernetes.io/ttl': '0',
                              'projectcalico.org/IPv4Address': '172.16.0.104/21',
                              'projectcalico.org/IPv4IPIPTunnelAddr': '192.168.48.128',
                              'volumes.kubernetes.io/controller-managed-attach-detach': 'true'},
              'cluster_name': None,
              'creation_timestamp': datetime.datetime(2019, 11, 6, 14, 14, 12, tzinfo=tzlocal()),
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': {'beta.kubernetes.io/arch': 'amd64',
                         'beta.kubernetes.io/os': 'linux',
                         'kubernetes.io/arch': 'amd64',
                         'kubernetes.io/hostname': 'tools-k8s-control-1',
                         'kubernetes.io/os': 'linux',
                         'node-role.kubernetes.io/master': ''},
              'managed_fields': None,
              'name': 'tools-k8s-control-1',
              'namespace': None,
              'owner_references': None,
              'resource_version': '11513770',
              'self_link': '/api/v1/nodes/tools-k8s-control-1',
              'uid': 'ea6201b3-0719-4779-a9ed-d11ba1d4101f'},
 'spec': {'config_source': None,
          'external_id': None,
          'pod_cidr': '192.168.0.0/24',
          'provider_id': None,
          'taints': [{'effect': 'NoSchedule',
                      'key': 'node-role.kubernetes.io/master',
                      'time_added': None,
                      'value': None}],
          'unschedulable': None},
 'status': {'addresses': [{'address': '172.16.0.104', 'type': 'InternalIP'},
                          {'address': 'tools-k8s-control-1',
                           'type': 'Hostname'}],
            'allocatable': {'cpu': '2',
                            'ephemeral-storage': '17951053795',
                            'hugepages-1Gi': '0',
                            'hugepages-2Mi': '0',
                            'memory': '3939056Ki',
                            'pods': '110'},
            'capacity': {'cpu': '2',
                         'ephemeral-storage': '19478140Ki',
                         'hugepages-1Gi': '0',
                         'hugepages-2Mi': '0',
                         'memory': '4041456Ki',
                         'pods': '110'},
            'conditions': [{'last_heartbeat_time': datetime.datetime(2019, 12, 13, 17, 17, 43, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 12, 13, 17, 17, 43, tzinfo=tzlocal()),
                            'message': 'Calico is running on this node',
                            'reason': 'CalicoIsUp',
                            'status': 'False',
                            'type': 'NetworkUnavailable'},
                           {'last_heartbeat_time': datetime.datetime(2020, 1, 5, 4, 53, 45, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 11, 6, 14, 14, 4, tzinfo=tzlocal()),
                            'message': 'kubelet has sufficient memory '
                                       'available',
                            'reason': 'KubeletHasSufficientMemory',
                            'status': 'False',
                            'type': 'MemoryPressure'},
                           {'last_heartbeat_time': datetime.datetime(2020, 1, 5, 4, 53, 45, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 11, 6, 14, 14, 4, tzinfo=tzlocal()),
                            'message': 'kubelet has no disk pressure',
                            'reason': 'KubeletHasNoDiskPressure',
                            'status': 'False',
                            'type': 'DiskPressure'},
                           {'last_heartbeat_time': datetime.datetime(2020, 1, 5, 4, 53, 45, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 11, 6, 14, 14, 4, tzinfo=tzlocal()),
                            'message': 'kubelet has sufficient PID available',
                            'reason': 'KubeletHasSufficientPID',
                            'status': 'False',
                            'type': 'PIDPressure'},
                           {'last_heartbeat_time': datetime.datetime(2020, 1, 5, 4, 53, 45, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 12, 13, 17, 16, 24, tzinfo=tzlocal()),
                            'message': 'kubelet is posting ready status. '
                                       'AppArmor enabled',
                            'reason': 'KubeletReady',
                            'status': 'True',
                            'type': 'Ready'}],
            'config': None,
            'daemon_endpoints': {'kubelet_endpoint': {'port': 10250}},
            'images': [{'names': ['k8s.gcr.io/kube-apiserver@sha256:b50e135bec86da5378ba2f8852d5bb966098d34abcb16510e36150d7b7dfd7b1',
                                  'k8s.gcr.io/kube-apiserver:v1.15.6'],
                        'size_bytes': 206880670},
                       {'names': ['k8s.gcr.io/kube-apiserver@sha256:304a1c38707834062ee87df62ef329d52a8b9a3e70459565d0a396479073f54c',
                                  'k8s.gcr.io/kube-apiserver:v1.15.1'],
                        'size_bytes': 206827454},
                       {'names': ['k8s.gcr.io/kube-controller-manager@sha256:1a1ccd6546b2149f3ec8ea42608046bdf74da7dc4d46e09e78f832060425b26d',
                                  'k8s.gcr.io/kube-controller-manager:v1.15.6'],
                        'size_bytes': 158775806},
                       {'names': ['k8s.gcr.io/kube-controller-manager@sha256:9abae95e428e228fe8f6d1630d55e79e018037460f3731312805c0f37471e4bf',
                                  'k8s.gcr.io/kube-controller-manager:v1.15.1'],
                        'size_bytes': 158722622},
                       {'names': ['calico/node@sha256:6679ccc9f19dba3eb084db991c788dc9661ad3b5d5bafaa3379644229dca6b05',
                                  'calico/node:v3.8.0'],
                        'size_bytes': 154769666},
                       {'names': ['calico/cni@sha256:decba0501ab0658e6e7da2f5625f1eabb8aba5690f9206caba3bf98caca5094c',
                                  'calico/cni:v3.8.0'],
                        'size_bytes': 143333412},
                       {'names': ['k8s.gcr.io/kube-proxy@sha256:ef245ddefef697c8b42611c237603acf41bfdb2b7ec3e434b7c3592864dcfff8',
                                  'k8s.gcr.io/kube-proxy:v1.15.6'],
                        'size_bytes': 82412380},
                       {'names': ['k8s.gcr.io/kube-proxy@sha256:08186f4897488e96cb098dd8d1d931af9a6ea718bb8737bf44bb76e42075f0ce',
                                  'k8s.gcr.io/kube-proxy:v1.15.1'],
                        'size_bytes': 82408284},
                       {'names': ['k8s.gcr.io/kube-scheduler@sha256:73b26c3ab2b80920196b723d86f7a8f698026bfae9808edcec9f1a8b588f30f1',
                                  'k8s.gcr.io/kube-scheduler:v1.15.6'],
                        'size_bytes': 81111678},
                       {'names': ['k8s.gcr.io/kube-scheduler@sha256:d0ee18a9593013fbc44b1920e4930f29b664b59a3958749763cb33b57e0e8956',
                                  'k8s.gcr.io/kube-scheduler:v1.15.1'],
                        'size_bytes': 81107582},
                       {'names': ['calico/kube-controllers@sha256:cf461efd25ee74d1855e1ee26db98fe87de00293f7d039212adb03c91fececcd',
                                  'calico/kube-controllers:v3.8.0'],
                        'size_bytes': 46809241},
                       {'names': ['k8s.gcr.io/coredns@sha256:02382353821b12c21b062c59184e227e001079bb13ebd01f9d3270ba0fcbf1e4',
                                  'k8s.gcr.io/coredns:1.3.1'],
                        'size_bytes': 40303560},
                       {'names': ['calico/pod2daemon-flexvol@sha256:6ec8b823e5ce3440318edfcdd2ab8b6660110782713f24f53dac5a3c227afb11',
                                  'calico/pod2daemon-flexvol:v3.8.0'],
                        'size_bytes': 9371181},
                       {'names': ['docker-registry.tools.wmflabs.org/pause@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610',
                                  'k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea',
                                  'docker-registry.tools.wmflabs.org/pause:3.1',
                                  'k8s.gcr.io/pause:3.1'],
                        'size_bytes': 742472}],
            'node_info': {'architecture': 'amd64',
                          'boot_id': '4537545d-3773-48bb-95f2-93e6b6e96a43',
                          'container_runtime_version': 'docker://19.3.4',
                          'kernel_version': '4.19.0-6-amd64',
                          'kube_proxy_version': 'v1.15.6',
                          'kubelet_version': 'v1.15.6',
                          'machine_id': '9cae425e02d34a48aa4142b6159aeced',
                          'operating_system': 'linux',
                          'os_image': 'Debian GNU/Linux 10 (buster)',
                          'system_uuid': '9cae425e-02d3-4a48-aa41-42b6159aeced'},
            'phase': None,
            'volumes_attached': None,
            'volumes_in_use': None}}
>>>