Page MenuHomePhabricator
Create Task
Maniphest T241381

Allow kubernetes observer role to read "nodes" data
Closed, ResolvedPublic
Actions

Description

Follow up to T233372: Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection. I would like the k8s-status tool to show the number of nodes in the cluster and possibly some data on how much CPU/RAM is in use.

>>> import kubernetes
>>> kubernetes.config.load_incluster_config()
>>> v1 = kubernetes.client.CoreV1Api()
>>> v1.list_node()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13267, in list_node
    (data) = self.list_node_with_http_info(**kwargs)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 13361, in list_node_with_http_info
    collection_formats=collection_formats)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 355, in request
    headers=headers)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in GET
    query_params=query_params)
  File "/data/project/k8s-status/www/python/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 23 Dec 2019 18:01:23 GMT', 'Content-Length': '292'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes is forbidden: User \"system:serviceaccount:tool-k8s-status:k8s-status-obs\" cannot list resource \"nodes\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"nodes"},"code":403}

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
toolforge-k8s: renamed files don't get changedoperations/puppetproduction+6 -1
toolforge-k8s: switch cluster monitor script to using a new clusterroleoperations/puppetproduction+106 -1
Customize query in gerrit

Related Objects

Event Timeline

bd808 created this task.Dec 23 2019, 6:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 23 2019, 6:45 PM
bd808 triaged this task as Medium priority.Dec 23 2019, 6:45 PM
bd808 added a comment.Dec 23 2019, 6:49 PM

I think this would be fixed by adding this snippet to clusterrole "view" or making a new clusterrole of our own including it and the existing "view" permissions:

- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
• Bstorm added a comment.Jan 2 2020, 3:41 PM

I strongly favor using our own clusterrole to preserve expectations (and changes between upgrades/rebuilds). I should have known that wouldn't be available to a system default read-only clusterrole :)

The role should be added to the base psp/roles we create at spin-up, and then the script can be updated to use that clusterrole instead.

• Bstorm added a comment.EditedJan 2 2020, 6:06 PM

@bd808: I was constructing a custom clusterrole when I noticed that you *do* have node metrics in view thanks to aggregation, which is likely to get you what you want anyway.

It includes:

- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch

The only unfortunate thing about this is that the API isn't included in the kubernetes client so you have to do this: https://github.com/kubernetes-client/python/issues/474

That would give you the same info as kubectl top nodes and kubectl top pods commands.

• Bstorm added a comment.Jan 2 2020, 6:11 PM

Specifically this is what I think is a way to use that https://github.com/kubernetes-client/python/issues/474#issuecomment-519901390

• Bstorm added a comment.Jan 2 2020, 6:30 PM

This is similar to what I expect that to get you:

root@toolsbeta-test-k8s-control-1:~# kubectl top nodes
NAME                           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
toolsbeta-test-k8s-control-1   165m         8%     1525Mi          39%       
toolsbeta-test-k8s-control-2   214m         10%    1651Mi          42%       
toolsbeta-test-k8s-control-3   161m         8%     1582Mi          41%       
toolsbeta-test-k8s-worker-1    878m         43%    2437Mi          63%       
toolsbeta-test-k8s-worker-2    436m         21%    1340Mi          34%       
toolsbeta-test-k8s-worker-3    316m         7%     3992Mi          50%

Please lemme know if I am wrong.

bd808 added a comment.Jan 2 2020, 8:21 PM

There is quite a bit of data in the metrics.k8s.io APIs. I will work on adding them into the app and see if that gets us all the way to what we are wanting for visualization right now.

>>> import kubernetes
>>> kubernetes.config.load_incluster_config()
>>> cust = kubernetes.client.CustomObjectsApi()
>>> import pprint
>>> pprint.pprint(cust.list_cluster_custom_object('metrics.k8s.io', 'v1beta1', 'nodes'))
{'apiVersion': 'metrics.k8s.io/v1beta1',
 'items': [{'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-4',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-4'},
            'timestamp': '2020-01-02T20:15:41Z',
            'usage': {'cpu': '115889991n', 'memory': '2202572Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-5',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-5'},
            'timestamp': '2020-01-02T20:15:43Z',
            'usage': {'cpu': '435133527n', 'memory': '1670660Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-control-1',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-control-1'},
            'timestamp': '2020-01-02T20:15:46Z',
            'usage': {'cpu': '139717252n', 'memory': '2029708Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-control-2',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-control-2'},
            'timestamp': '2020-01-02T20:15:41Z',
            'usage': {'cpu': '109962548n', 'memory': '1695480Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-control-3',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-control-3'},
            'timestamp': '2020-01-02T20:15:42Z',
            'usage': {'cpu': '109308931n', 'memory': '1693756Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-1',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-1'},
            'timestamp': '2020-01-02T20:15:41Z',
            'usage': {'cpu': '131675178n', 'memory': '1576956Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-2',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-2'},
            'timestamp': '2020-01-02T20:15:46Z',
            'usage': {'cpu': '111128381n', 'memory': '1518384Ki'},
            'window': '30s'},
           {'metadata': {'creationTimestamp': '2020-01-02T20:16:42Z',
                         'name': 'tools-k8s-worker-3',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes/tools-k8s-worker-3'},
            'timestamp': '2020-01-02T20:15:48Z',
            'usage': {'cpu': '186817490n', 'memory': '2050004Ki'},
            'window': '30s'}],
 'kind': 'NodeMetricsList',
 'metadata': {'selfLink': '/apis/metrics.k8s.io/v1beta1/nodes'}}
>>> pprint.pprint(cust.list_cluster_custom_object('metrics.k8s.io', 'v1beta1', 'pods'))
{'apiVersion': 'metrics.k8s.io/v1beta1',
 'items': [{'containers': [{'name': 'coredns',
                            'usage': {'cpu': '3382319n', 'memory': '23148Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'coredns-5c98db65d4-hq6hc',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-5c98db65d4-hq6hc'},
            'timestamp': '2020-01-02T20:17:33Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '699798n', 'memory': '21768Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-wkgzb',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-wkgzb'},
            'timestamp': '2020-01-02T20:17:31Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '433690n', 'memory': '20208Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-77qj4',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-77qj4'},
            'timestamp': '2020-01-02T20:17:46Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '4230284n', 'memory': '20148Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-8sl46',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-8sl46'},
            'timestamp': '2020-01-02T20:17:38Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '25287162n',
                                      'memory': '219160Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-qz4tn',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-qz4tn'},
            'timestamp': '2020-01-02T20:17:40Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '387686n', 'memory': '17176Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-7ssq4',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-7ssq4'},
            'timestamp': '2020-01-02T20:17:44Z',
            'window': '30s'},
           {'containers': [{'name': 'webhook',
                            'usage': {'cpu': '18097n', 'memory': '4956Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'ingress-admission-55fb8554b5-btp7h',
                         'namespace': 'ingress-admission',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-admission/pods/ingress-admission-55fb8554b5-btp7h'},
            'timestamp': '2020-01-02T20:17:42Z',
            'window': '30s'},
           {'containers': [{'name': 'webhook',
                            'usage': {'cpu': '27464n', 'memory': '5744Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'registry-admission-6f5f6589c5-tzzk5',
                         'namespace': 'registry-admission',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/registry-admission/pods/registry-admission-6f5f6589c5-tzzk5'},
            'timestamp': '2020-01-02T20:17:47Z',
            'window': '30s'},
           {'containers': [{'name': 'nginx-ingress-controller',
                            'usage': {'cpu': '3586290n',
                                      'memory': '171204Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'nginx-ingress-64dc7c9c57-2j7qd',
                         'namespace': 'ingress-nginx',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-nginx/pods/nginx-ingress-64dc7c9c57-2j7qd'},
            'timestamp': '2020-01-02T20:17:39Z',
            'window': '30s'},
           {'containers': [{'name': 'nginx-ingress-controller',
                            'usage': {'cpu': '12302417n',
                                      'memory': '166668Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'nginx-ingress-64dc7c9c57-hsxkj',
                         'namespace': 'ingress-nginx',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-nginx/pods/nginx-ingress-64dc7c9c57-hsxkj'},
            'timestamp': '2020-01-02T20:17:41Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '66255n', 'memory': '207468Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'k8s-status-7858fc66cd-6t2tg',
                         'namespace': 'tool-k8s-status',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-k8s-status/pods/k8s-status-7858fc66cd-6t2tg'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'maintain-kubeusers',
                            'usage': {'cpu': '76121n', 'memory': '23904Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'maintain-kubeusers-ops',
                         'namespace': 'maintain-kubeusers',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/maintain-kubeusers/pods/maintain-kubeusers-ops'},
            'timestamp': '2020-01-02T20:17:46Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '147589n', 'memory': '36468Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'bd808-test-6b88576f69-qttbn',
                         'namespace': 'tool-bd808-test',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-bd808-test/pods/bd808-test-6b88576f69-qttbn'},
            'timestamp': '2020-01-02T20:17:47Z',
            'window': '30s'},
           {'containers': [{'name': 'webhook',
                            'usage': {'cpu': '13916n', 'memory': '4924Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'ingress-admission-55fb8554b5-6c98v',
                         'namespace': 'ingress-admission',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-admission/pods/ingress-admission-55fb8554b5-6c98v'},
            'timestamp': '2020-01-02T20:17:40Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-apiserver',
                            'usage': {'cpu': '18771837n',
                                      'memory': '365324Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-apiserver-tools-k8s-control-3',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-apiserver-tools-k8s-control-3'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '18705688n',
                                      'memory': '220284Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-44ntg',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-44ntg'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'bot',
                            'usage': {'cpu': '966472n', 'memory': '58592Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'stashbot.bot-5bbf56f756-pgxmn',
                         'namespace': 'tool-stashbot',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-stashbot/pods/stashbot.bot-5bbf56f756-pgxmn'},
            'timestamp': '2020-01-02T20:17:48Z',
            'window': '30s'},
           {'containers': [{'name': 'interactive',
                            'usage': {'cpu': '0', 'memory': '48956Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'interactive',
                         'namespace': 'tool-k8s-status',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-k8s-status/pods/interactive'},
            'timestamp': '2020-01-02T20:17:37Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-scheduler',
                            'usage': {'cpu': '765u', 'memory': '12500Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-scheduler-tools-k8s-control-3',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-tools-k8s-control-3'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'metrics-server',
                            'usage': {'cpu': '738180n', 'memory': '23584Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'metrics-server-575d5f6d95-ff7pp',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/metrics-server-575d5f6d95-ff7pp'},
            'timestamp': '2020-01-02T20:17:39Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-controller-manager',
                            'usage': {'cpu': '407090n', 'memory': '13308Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-controller-manager-tools-k8s-control-3',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-controller-manager-tools-k8s-control-3'},
            'timestamp': '2020-01-02T20:17:46Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '4566727n', 'memory': '19984Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-qvfvp',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-qvfvp'},
            'timestamp': '2020-01-02T20:17:43Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-kube-controllers',
                            'usage': {'cpu': '1380455n', 'memory': '18504Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-kube-controllers-59f54d6bbc-mwrjm',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-kube-controllers-59f54d6bbc-mwrjm'},
            'timestamp': '2020-01-02T20:17:36Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '16508472n',
                                      'memory': '204652Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-c8rxg',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-c8rxg'},
            'timestamp': '2020-01-02T20:17:48Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-scheduler',
                            'usage': {'cpu': '562934n', 'memory': '13180Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-scheduler-tools-k8s-control-2',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-tools-k8s-control-2'},
            'timestamp': '2020-01-02T20:17:34Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '31960738n',
                                      'memory': '215788Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-snn29',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-snn29'},
            'timestamp': '2020-01-02T20:17:32Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '24848227n',
                                      'memory': '213428Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-nrk46',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-nrk46'},
            'timestamp': '2020-01-02T20:17:44Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-controller-manager',
                            'usage': {'cpu': '353369n', 'memory': '12744Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-controller-manager-tools-k8s-control-2',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-controller-manager-tools-k8s-control-2'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '354084n', 'memory': '17128Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-8fv9z',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-8fv9z'},
            'timestamp': '2020-01-02T20:17:46Z',
            'window': '30s'},
           {'containers': [{'name': 'maintain-kubeusers',
                            'usage': {'cpu': '350151364n',
                                      'memory': '63260Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'maintain-kubeusers-6d9c45f4bc-5bqq5',
                         'namespace': 'maintain-kubeusers',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/maintain-kubeusers/pods/maintain-kubeusers-6d9c45f4bc-5bqq5'},
            'timestamp': '2020-01-02T20:17:47Z',
            'window': '30s'},
           {'containers': [{'name': 'nginx-ingress-controller',
                            'usage': {'cpu': '3229161n',
                                      'memory': '202184Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'nginx-ingress-64dc7c9c57-twsvk',
                         'namespace': 'ingress-nginx',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/ingress-nginx/pods/nginx-ingress-64dc7c9c57-twsvk'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-apiserver',
                            'usage': {'cpu': '152427205n',
                                      'memory': '299496Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-apiserver-tools-k8s-control-2',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-apiserver-tools-k8s-control-2'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '163393n', 'memory': '35128Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'replag-d9b46cb69-gjdkq',
                         'namespace': 'tool-replag',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-replag/pods/replag-d9b46cb69-gjdkq'},
            'timestamp': '2020-01-02T20:17:47Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-state-metrics',
                            'usage': {'cpu': '1603192n', 'memory': '40536Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-state-metrics-7678f4fc5c-59627',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-state-metrics-7678f4fc5c-59627'},
            'timestamp': '2020-01-02T20:17:34Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '18829429n',
                                      'memory': '185148Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-g64gn',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-g64gn'},
            'timestamp': '2020-01-02T20:17:36Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '21814334n',
                                      'memory': '213800Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-pcmlx',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-pcmlx'},
            'timestamp': '2020-01-02T20:17:43Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-apiserver',
                            'usage': {'cpu': '29630223n',
                                      'memory': '353296Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-apiserver-tools-k8s-control-1',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-apiserver-tools-k8s-control-1'},
            'timestamp': '2020-01-02T20:17:48Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-scheduler',
                            'usage': {'cpu': '1511734n', 'memory': '14828Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-scheduler-tools-k8s-control-1',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-tools-k8s-control-1'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '505140n', 'memory': '27020Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-wwmjc',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-wwmjc'},
            'timestamp': '2020-01-02T20:17:44Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-controller-manager',
                            'usage': {'cpu': '9742817n',
                                      'memory': '231576Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-controller-manager-tools-k8s-control-1',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-controller-manager-tools-k8s-control-1'},
            'timestamp': '2020-01-02T20:17:41Z',
            'window': '30s'},
           {'containers': [{'name': 'kube-proxy',
                            'usage': {'cpu': '412237n', 'memory': '22764Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'kube-proxy-r94mq',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-r94mq'},
            'timestamp': '2020-01-02T20:17:41Z',
            'window': '30s'},
           {'containers': [{'name': 'calico-node',
                            'usage': {'cpu': '18771922n',
                                      'memory': '216352Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'calico-node-dn2pk',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-dn2pk'},
            'timestamp': '2020-01-02T20:17:45Z',
            'window': '30s'},
           {'containers': [{'name': 'coredns',
                            'usage': {'cpu': '3255301n', 'memory': '24632Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'coredns-5c98db65d4-97hjp',
                         'namespace': 'kube-system',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-5c98db65d4-97hjp'},
            'timestamp': '2020-01-02T20:17:35Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '132148n', 'memory': '29368Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'stashbot-85cc8c6c55-lz54v',
                         'namespace': 'tool-stashbot',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-stashbot/pods/stashbot-85cc8c6c55-lz54v'},
            'timestamp': '2020-01-02T20:17:42Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '53184n', 'memory': '45156Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'bash-666c74f75f-4zsc6',
                         'namespace': 'tool-bash',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-bash/pods/bash-666c74f75f-4zsc6'},
            'timestamp': '2020-01-02T20:17:39Z',
            'window': '30s'},
           {'containers': [{'name': 'webservice',
                            'usage': {'cpu': '76570007n',
                                      'memory': '205900Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'fourohfour-787f7f88f9-npgx5',
                         'namespace': 'tool-fourohfour',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/tool-fourohfour/pods/fourohfour-787f7f88f9-npgx5'},
            'timestamp': '2020-01-02T20:17:33Z',
            'window': '30s'},
           {'containers': [{'name': 'webhook',
                            'usage': {'cpu': '221956n', 'memory': '7632Ki'}}],
            'metadata': {'creationTimestamp': '2020-01-02T20:18:03Z',
                         'name': 'registry-admission-6f5f6589c5-clj2w',
                         'namespace': 'registry-admission',
                         'selfLink': '/apis/metrics.k8s.io/v1beta1/namespaces/registry-admission/pods/registry-admission-6f5f6589c5-clj2w'},
            'timestamp': '2020-01-02T20:17:40Z',
            'window': '30s'}],
 'kind': 'PodMetricsList',
 'metadata': {'selfLink': '/apis/metrics.k8s.io/v1beta1/pods'}}
• Bstorm added a comment.Jan 2 2020, 10:45 PM

The fact that it reports CPU in "nanoCPUs"... Apparently someone thinks that's how finely you need to slice this kind of thing.

bd808 added a comment.Jan 3 2020, 5:51 AM

I have added some data from the metrics api to https://tools.wmflabs.org/k8s-status/. This does allow us to show:

  • total number of control nodes
  • total number of worker nodes
  • ram in use
  • cpu in use

It does not let us show ram & cpu available which I think would be interesting. Exposing the node API would give access to a lot of metadata about each node, including the allocatable CPU and RAM:

$ kubectl get nodes tools-k8s-worker-4 -o yaml
apiVersion: v1
kind: Node
metadata:
  annotations:
    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
    node.alpha.kubernetes.io/ttl: "0"
    projectcalico.org/IPv4Address: 172.16.0.113/21
    projectcalico.org/IPv4IPIPTunnelAddr: 192.168.81.0
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2019-11-29T12:12:34Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: tools-k8s-worker-4
    kubernetes.io/os: linux
  name: tools-k8s-worker-4
  resourceVersion: "11111196"
  selfLink: /api/v1/nodes/tools-k8s-worker-4
  uid: e130408d-c677-4a53-94ac-ae58aa1976c5
spec:
  podCIDR: 192.168.6.0/24
status:
  addresses:
  - address: 172.16.0.113
    type: InternalIP
  - address: tools-k8s-worker-4
    type: Hostname
  allocatable:
    cpu: "4"
    ephemeral-storage: "17951053795"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 8067460Ki
    pods: "110"
  capacity:
    cpu: "4"
    ephemeral-storage: 19478140Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 8169860Ki
    pods: "110"
  conditions:
  - lastHeartbeatTime: "2019-12-12T12:19:19Z"
    lastTransitionTime: "2019-12-12T12:19:19Z"
    message: Calico is running on this node
    reason: CalicoIsUp
    status: "False"
    type: NetworkUnavailable
  - lastHeartbeatTime: "2020-01-03T05:40:15Z"
    lastTransitionTime: "2019-11-29T12:12:34Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2020-01-03T05:40:15Z"
    lastTransitionTime: "2019-11-29T12:12:34Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2020-01-03T05:40:15Z"
    lastTransitionTime: "2019-11-29T12:12:34Z"
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: "2020-01-03T05:40:15Z"
    lastTransitionTime: "2019-11-29T12:12:44Z"
    message: kubelet is posting ready status. AppArmor enabled
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-jdk11-sssd-web@sha256:0d397ff6c41a29426c059e5b3c2f07303c27d000c853719c94222869af9ab734
    - docker-registry.tools.wmflabs.org/toolforge-jdk11-sssd-web:latest
    sizeBytes: 1059181798
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web@sha256:ecf1b20d28eac60c415e6c8b815b9c3e140ee56b4a7462ca6a0936bc62142713
    sizeBytes: 919889321
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-golang-sssd-web@sha256:32cd42bc997ca1ba6e1e64accb5363bb0116ad9e55e4e2aa9ff4b23d25494633
    - docker-registry.tools.wmflabs.org/toolforge-golang-sssd-web:latest
    sizeBytes: 891721425
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base@sha256:25f796e556a62b20e712f23492df89102a2680933bada363ad7a8a974f0e944a
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base:latest
    sizeBytes: 886712537
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python35-sssd-web@sha256:5bd9dab68c52e11be793f146f27f22d63fb6c41fe084ad4a0734d3bc41287867
    - docker-registry.tools.wmflabs.org/toolforge-python35-sssd-web:latest
    sizeBytes: 836890582
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python35-sssd-web@sha256:5de157b4ec83c060eafd985cdb715242d0beed8f15a30d9fe29aeae617cca3c1
    sizeBytes: 836865747
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-golang111-sssd-web@sha256:f27a711eca47af1566c2ecdc663bac40010adea7b368a4be543534e4428fb7da
    - docker-registry.tools.wmflabs.org/toolforge-golang111-sssd-web:latest
    sizeBytes: 833649533
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-node10-sssd-web@sha256:c726accc1ff69d0aab0d6e7c953aa8a247dbf2adcbed4875237193ab4220c5cb
    - docker-registry.tools.wmflabs.org/toolforge-node10-sssd-web:latest
    sizeBytes: 784062125
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web@sha256:25bb8ecc642f4d52e4403b39c00fddb7a2abbaa9488c7ac0db8116f3cb57b0e9
    - docker-registry.tools.wmflabs.org/toolforge-python37-sssd-web:latest
    sizeBytes: 767916282
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-jdk8-sssd-web@sha256:114c594cf54584a2ce550511340efb6b4534a6774b3a74fb06f6d5c3912b3471
    - docker-registry.tools.wmflabs.org/toolforge-jdk8-sssd-web:latest
    sizeBytes: 739886916
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-ruby25-sssd-web@sha256:444f4fc416c589b027d340799b014d3f72a75b64ca6bd38145bb745495535efb
    - docker-registry.tools.wmflabs.org/toolforge-ruby25-sssd-web:latest
    sizeBytes: 736192302
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python34-sssd-web@sha256:24c92be4136258b1a0ad810184e0a8956e930cb9be0aee93fa2d34cc8953ff6a
    - docker-registry.tools.wmflabs.org/toolforge-python34-sssd-web:latest
    sizeBytes: 713162449
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-python2-sssd-web@sha256:b3dbbd87d2b1c21491db568894278ab866cc390f9ecaa9768cbbc55628cc8ea9
    - docker-registry.tools.wmflabs.org/toolforge-python2-sssd-web:latest
    sizeBytes: 685182640
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-node6-sssd-web@sha256:49b2df0004e0b4bec1da29462b54a882c6dc15d315fa90b935966cb5b6437a5d
    - docker-registry.tools.wmflabs.org/toolforge-node6-sssd-web:latest
    sizeBytes: 614698538
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web@sha256:7454a32bf6ca360dd56b3e4513b6329cbc08a192faa1c015afab88ddbf12d77a
    - docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest
    sizeBytes: 591205517
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-ruby21-sssd-web@sha256:91695f4e72b00d16f2cb0105bc0507bcce428c86dbfe0be17f00cbaae0118785
    - docker-registry.tools.wmflabs.org/toolforge-ruby21-sssd-web:latest
    sizeBytes: 551482292
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-php72-sssd-web@sha256:7fa16aeaa23b44d550059944749d7df47558b46257d5bdd98d29da1d67604d5e
    - docker-registry.tools.wmflabs.org/toolforge-php72-sssd-web:latest
    sizeBytes: 538009244
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-php5-sssd-web@sha256:3d293a785228cdef5e0f34592737126a6f70b014f8c6ed8e57c4f6da58f56075
    - docker-registry.tools.wmflabs.org/toolforge-php5-sssd-web:latest
    sizeBytes: 531993968
  - names:
    - docker-registry.tools.wmflabs.org/nginx-ingress-controller@sha256:b9cd638b8849f25210740b075d27ef2e55ffd2861488ead98276aa70b8a859ab
    - docker-registry.tools.wmflabs.org/nginx-ingress-controller:0.25.1
    sizeBytes: 510687563
  - names:
    - docker-registry.tools.wmflabs.org/toolforge-tcl86-sssd-web@sha256:a35c12693cc3991ccb646670a56108f64c12240a91f110d50618f6cd05fc5f5e
    - docker-registry.tools.wmflabs.org/toolforge-tcl86-sssd-web:latest
    sizeBytes: 469721313
  - names:
    - calico/node@sha256:6679ccc9f19dba3eb084db991c788dc9661ad3b5d5bafaa3379644229dca6b05
    - calico/node:v3.8.0
    sizeBytes: 154769666
  - names:
    - calico/cni@sha256:decba0501ab0658e6e7da2f5625f1eabb8aba5690f9206caba3bf98caca5094c
    - calico/cni:v3.8.0
    sizeBytes: 143333412
  - names:
    - k8s.gcr.io/kube-proxy@sha256:ef245ddefef697c8b42611c237603acf41bfdb2b7ec3e434b7c3592864dcfff8
    - k8s.gcr.io/kube-proxy:v1.15.6
    sizeBytes: 82412380
  - names:
    - quay.io/coreos/kube-state-metrics@sha256:f75c3e5c5c7f65846ddd6883d6187b38f77721a3938f241c9e5d0ebe7beb8e19
    - quay.io/coreos/kube-state-metrics:v1.8.0
    sizeBytes: 33616576
  - names:
    - calico/pod2daemon-flexvol@sha256:6ec8b823e5ce3440318edfcdd2ab8b6660110782713f24f53dac5a3c227afb11
    - calico/pod2daemon-flexvol:v3.8.0
    sizeBytes: 9371181
  - names:
    - docker-registry.tools.wmflabs.org/pause@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610
    - docker-registry.tools.wmflabs.org/pause:3.1
    sizeBytes: 742472
  nodeInfo:
    architecture: amd64
    bootID: 6c6134fa-b225-46d7-98f1-72132e34bce2
    containerRuntimeVersion: docker://19.3.5
    kernelVersion: 4.19.0-6-amd64
    kubeProxyVersion: v1.15.6
    kubeletVersion: v1.15.6
    machineID: 90c9b8c4c3ac49b38a17edd815457e2a
    operatingSystem: linux
    osImage: Debian GNU/Linux 10 (buster)
    systemUUID: 90c9b8c4-c3ac-49b3-8a17-edd815457e2a
• Bstorm added a comment.Jan 3 2020, 5:34 PM

I found me another reason I'd rather not use the "view" clusterrole anyway. Patch incoming :)

gerritbot added a comment.Jan 3 2020, 5:36 PM

Change 561889 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: switch cluster monitor script to using a new clusterrole

https://gerrit.wikimedia.org/r/561889

gerritbot added a project: Patch-For-Review.Jan 3 2020, 5:36 PM
gerritbot added a comment.Jan 3 2020, 5:42 PM

Change 561889 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: switch cluster monitor script to using a new clusterrole

https://gerrit.wikimedia.org/r/561889

gerritbot added a comment.Jan 3 2020, 6:02 PM

Change 561891 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-k8s: renamed files don't get changed

https://gerrit.wikimedia.org/r/561891

gerritbot added a comment.Jan 3 2020, 6:03 PM

Change 561891 merged by Bstorm:
[operations/puppet@production] toolforge-k8s: renamed files don't get changed

https://gerrit.wikimedia.org/r/561891

• Bstorm added a comment.Jan 3 2020, 6:18 PM
root@tools-k8s-control-1:~# kubectl delete clusterrolebindings k8s-status-obs
clusterrolebinding.rbac.authorization.k8s.io "k8s-status-obs" deleted
root@tools-k8s-control-1:~# wmcs-k8s-enable-cluster-monitor k8s-status
NAME              STATUS   AGE
tool-k8s-status   Active   17d
Creating the service account...
serviceaccount/k8s-status-obs unchanged
Enabling read-only access to the cluster...
clusterrolebinding.rbac.authorization.k8s.io/k8s-status-obs created
rolebinding.rbac.authorization.k8s.io/k8s-status-obs-psp unchanged
NAME             SECRETS   AGE
k8s-status-obs   1         14d
*********************
Done!

Ok, you should be using the new role now. Let me know how that works for you.

• Bstorm added a comment.Jan 3 2020, 6:20 PM

Notably, the tool is still working on the web, which is good 😁

bd808 closed this task as Resolved.Jan 5 2020, 4:56 AM
bd808 assigned this task to • Bstorm.

Works!

>>> import k8s.client
>>> v1 = k8s.client.corev1_client()
>>> v1.list_node().items[0]
{'api_version': None,
 'kind': None,
 'metadata': {'annotations': {'kubeadm.alpha.kubernetes.io/cri-socket': '/var/run/dockershim.sock',
                              'node.alpha.kubernetes.io/ttl': '0',
                              'projectcalico.org/IPv4Address': '172.16.0.104/21',
                              'projectcalico.org/IPv4IPIPTunnelAddr': '192.168.48.128',
                              'volumes.kubernetes.io/controller-managed-attach-detach': 'true'},
              'cluster_name': None,
              'creation_timestamp': datetime.datetime(2019, 11, 6, 14, 14, 12, tzinfo=tzlocal()),
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': {'beta.kubernetes.io/arch': 'amd64',
                         'beta.kubernetes.io/os': 'linux',
                         'kubernetes.io/arch': 'amd64',
                         'kubernetes.io/hostname': 'tools-k8s-control-1',
                         'kubernetes.io/os': 'linux',
                         'node-role.kubernetes.io/master': ''},
              'managed_fields': None,
              'name': 'tools-k8s-control-1',
              'namespace': None,
              'owner_references': None,
              'resource_version': '11513770',
              'self_link': '/api/v1/nodes/tools-k8s-control-1',
              'uid': 'ea6201b3-0719-4779-a9ed-d11ba1d4101f'},
 'spec': {'config_source': None,
          'external_id': None,
          'pod_cidr': '192.168.0.0/24',
          'provider_id': None,
          'taints': [{'effect': 'NoSchedule',
                      'key': 'node-role.kubernetes.io/master',
                      'time_added': None,
                      'value': None}],
          'unschedulable': None},
 'status': {'addresses': [{'address': '172.16.0.104', 'type': 'InternalIP'},
                          {'address': 'tools-k8s-control-1',
                           'type': 'Hostname'}],
            'allocatable': {'cpu': '2',
                            'ephemeral-storage': '17951053795',
                            'hugepages-1Gi': '0',
                            'hugepages-2Mi': '0',
                            'memory': '3939056Ki',
                            'pods': '110'},
            'capacity': {'cpu': '2',
                         'ephemeral-storage': '19478140Ki',
                         'hugepages-1Gi': '0',
                         'hugepages-2Mi': '0',
                         'memory': '4041456Ki',
                         'pods': '110'},
            'conditions': [{'last_heartbeat_time': datetime.datetime(2019, 12, 13, 17, 17, 43, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 12, 13, 17, 17, 43, tzinfo=tzlocal()),
                            'message': 'Calico is running on this node',
                            'reason': 'CalicoIsUp',
                            'status': 'False',
                            'type': 'NetworkUnavailable'},
                           {'last_heartbeat_time': datetime.datetime(2020, 1, 5, 4, 53, 45, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 11, 6, 14, 14, 4, tzinfo=tzlocal()),
                            'message': 'kubelet has sufficient memory '
                                       'available',
                            'reason': 'KubeletHasSufficientMemory',
                            'status': 'False',
                            'type': 'MemoryPressure'},
                           {'last_heartbeat_time': datetime.datetime(2020, 1, 5, 4, 53, 45, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 11, 6, 14, 14, 4, tzinfo=tzlocal()),
                            'message': 'kubelet has no disk pressure',
                            'reason': 'KubeletHasNoDiskPressure',
                            'status': 'False',
                            'type': 'DiskPressure'},
                           {'last_heartbeat_time': datetime.datetime(2020, 1, 5, 4, 53, 45, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 11, 6, 14, 14, 4, tzinfo=tzlocal()),
                            'message': 'kubelet has sufficient PID available',
                            'reason': 'KubeletHasSufficientPID',
                            'status': 'False',
                            'type': 'PIDPressure'},
                           {'last_heartbeat_time': datetime.datetime(2020, 1, 5, 4, 53, 45, tzinfo=tzlocal()),
                            'last_transition_time': datetime.datetime(2019, 12, 13, 17, 16, 24, tzinfo=tzlocal()),
                            'message': 'kubelet is posting ready status. '
                                       'AppArmor enabled',
                            'reason': 'KubeletReady',
                            'status': 'True',
                            'type': 'Ready'}],
            'config': None,
            'daemon_endpoints': {'kubelet_endpoint': {'port': 10250}},
            'images': [{'names': ['k8s.gcr.io/kube-apiserver@sha256:b50e135bec86da5378ba2f8852d5bb966098d34abcb16510e36150d7b7dfd7b1',
                                  'k8s.gcr.io/kube-apiserver:v1.15.6'],
                        'size_bytes': 206880670},
                       {'names': ['k8s.gcr.io/kube-apiserver@sha256:304a1c38707834062ee87df62ef329d52a8b9a3e70459565d0a396479073f54c',
                                  'k8s.gcr.io/kube-apiserver:v1.15.1'],
                        'size_bytes': 206827454},
                       {'names': ['k8s.gcr.io/kube-controller-manager@sha256:1a1ccd6546b2149f3ec8ea42608046bdf74da7dc4d46e09e78f832060425b26d',
                                  'k8s.gcr.io/kube-controller-manager:v1.15.6'],
                        'size_bytes': 158775806},
                       {'names': ['k8s.gcr.io/kube-controller-manager@sha256:9abae95e428e228fe8f6d1630d55e79e018037460f3731312805c0f37471e4bf',
                                  'k8s.gcr.io/kube-controller-manager:v1.15.1'],
                        'size_bytes': 158722622},
                       {'names': ['calico/node@sha256:6679ccc9f19dba3eb084db991c788dc9661ad3b5d5bafaa3379644229dca6b05',
                                  'calico/node:v3.8.0'],
                        'size_bytes': 154769666},
                       {'names': ['calico/cni@sha256:decba0501ab0658e6e7da2f5625f1eabb8aba5690f9206caba3bf98caca5094c',
                                  'calico/cni:v3.8.0'],
                        'size_bytes': 143333412},
                       {'names': ['k8s.gcr.io/kube-proxy@sha256:ef245ddefef697c8b42611c237603acf41bfdb2b7ec3e434b7c3592864dcfff8',
                                  'k8s.gcr.io/kube-proxy:v1.15.6'],
                        'size_bytes': 82412380},
                       {'names': ['k8s.gcr.io/kube-proxy@sha256:08186f4897488e96cb098dd8d1d931af9a6ea718bb8737bf44bb76e42075f0ce',
                                  'k8s.gcr.io/kube-proxy:v1.15.1'],
                        'size_bytes': 82408284},
                       {'names': ['k8s.gcr.io/kube-scheduler@sha256:73b26c3ab2b80920196b723d86f7a8f698026bfae9808edcec9f1a8b588f30f1',
                                  'k8s.gcr.io/kube-scheduler:v1.15.6'],
                        'size_bytes': 81111678},
                       {'names': ['k8s.gcr.io/kube-scheduler@sha256:d0ee18a9593013fbc44b1920e4930f29b664b59a3958749763cb33b57e0e8956',
                                  'k8s.gcr.io/kube-scheduler:v1.15.1'],
                        'size_bytes': 81107582},
                       {'names': ['calico/kube-controllers@sha256:cf461efd25ee74d1855e1ee26db98fe87de00293f7d039212adb03c91fececcd',
                                  'calico/kube-controllers:v3.8.0'],
                        'size_bytes': 46809241},
                       {'names': ['k8s.gcr.io/coredns@sha256:02382353821b12c21b062c59184e227e001079bb13ebd01f9d3270ba0fcbf1e4',
                                  'k8s.gcr.io/coredns:1.3.1'],
                        'size_bytes': 40303560},
                       {'names': ['calico/pod2daemon-flexvol@sha256:6ec8b823e5ce3440318edfcdd2ab8b6660110782713f24f53dac5a3c227afb11',
                                  'calico/pod2daemon-flexvol:v3.8.0'],
                        'size_bytes': 9371181},
                       {'names': ['docker-registry.tools.wmflabs.org/pause@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610',
                                  'k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea',
                                  'docker-registry.tools.wmflabs.org/pause:3.1',
                                  'k8s.gcr.io/pause:3.1'],
                        'size_bytes': 742472}],
            'node_info': {'architecture': 'amd64',
                          'boot_id': '4537545d-3773-48bb-95f2-93e6b6e96a43',
                          'container_runtime_version': 'docker://19.3.4',
                          'kernel_version': '4.19.0-6-amd64',
                          'kube_proxy_version': 'v1.15.6',
                          'kubelet_version': 'v1.15.6',
                          'machine_id': '9cae425e02d34a48aa4142b6159aeced',
                          'operating_system': 'linux',
                          'os_image': 'Debian GNU/Linux 10 (buster)',
                          'system_uuid': '9cae425e-02d3-4a48-aa41-42b6159aeced'},
            'phase': None,
            'volumes_attached': None,
            'volumes_in_use': None}}
>>>
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits