PAWS pods hung in state "Terminating"
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• madhuvishy
	Jan 9 2018, 4:47 PM

Description

These pods are stuck in state Terminating and I can't even force kill them with delete --force.

yuvipanda@tools-paws-master-01:~$ kubectl --namespace=prod get pod | grep Terminating
jupyter-aftabbot                           1/1       Terminating   0          43d
jupyter-akohdonemma                        1/1       Terminating   1          38d
jupyter-discasto                           1/1       Terminating   0          46d
jupyter-doc-5ftaxon                        1/1       Terminating   1          47d
jupyter-mattho69                           1/1       Terminating   0          38d

The logs for these pods look like:

[E 2018-01-07 07:05:17.747 SingleUserNotebookApp log:122] 500 GET /paws/user/discasto/api/kernels/598ee897-cb1b-49b6-8e17-596dd14f1d59/channels?session_id=67C32F4F44314ABABEA79C556F36CAE5 (@10.244.5.110) 9435.98ms
[W 2018-01-07 07:05:28.308 SingleUserNotebookApp handlers:257] Replacing stale connection: bfce21c9-5094-446d-8213-662b31164575:9E9E5B7E8A1542A0829B98393D5EC68E
[W 2018-01-07 07:05:45.552 SingleUserNotebookApp handlers:257] Replacing stale connection: 165e4119-78cc-4d1c-83b4-0b6fb23a12ac:8081963C394947C282587F3E3059CC49
[W 2018-01-07 07:06:00.139 SingleUserNotebookApp handlers:257] Replacing stale connection: 598ee897-cb1b-49b6-8e17-596dd14f1d59:67C32F4F44314ABABEA79C556F36CAE5
[W 2018-01-07 07:06:14.510 SingleUserNotebookApp handlers:257] Replacing stale connection: bfce21c9-5094-446d-8213-662b31164575:9E9E5B7E8A1542A0829B98393D5EC68E
[W 2018-01-07 07:06:30.504 SingleUserNotebookApp handlers:257] Replacing stale connection: 165e4119-78cc-4d1c-83b4-0b6fb23a12ac:8081963C394947C282587F3E3059CC49
[W 2018-01-07 07:07:05.454 SingleUserNotebookApp handlers:257] Replacing stale connection: 2d4fda05-9deb-4d55-95fa-74d116f419b8:0A1BBD0DDBEE45E9957C476D51D8F3B7
[W 2018-01-07 07:07:46.070 SingleUserNotebookApp handlers:257] Replacing stale connection: 598ee897-cb1b-49b6-8e17-596dd14f1d59:67C32F4F44314ABABEA79C556F36CAE5
[W 2018-01-07 07:07:57.094 SingleUserNotebookApp handlers:257] Replacing stale connection: 165e4119-78cc-4d1c-83b4-0b6fb23a12ac:8081963C394947C282587F3E3059CC49
[W 2018-01-07 07:07:57.156 SingleUserNotebookApp handlers:257] Replacing stale connection: 2d4fda05-9deb-4d55-95fa-74d116f419b8:0A1BBD0DDBEE45E9957C476D51D8F3B7
[W 2018-01-07 07:07:57.158 SingleUserNotebookApp handlers:257] Replacing stale connection: 598ee897-cb1b-49b6-8e17-596dd14f1d59:67C32F4F44314ABABEA79C556F36CAE5
[E 2018-01-07 07:07:57.241 SingleUserNotebookApp ioloop:638] Exception in callback <bound method WebSocketMixin.send_ping of ZMQChannelsHandler(165e4119-78cc-4d1c-83b4-0b6fb23a12ac)>
    Traceback (most recent call last):
      File "/srv/paws/lib/python3.6/site-packages/tornado/ioloop.py", line 1026, in _run
        return self.callback()
      File "/srv/paws/lib/python3.6/site-packages/notebook/base/zmqhandlers.py", line 197, in send_ping
        self.ping(b'')
      File "/srv/paws/lib/python3.6/site-packages/tornado/websocket.py", line 314, in ping
        raise WebSocketClosedError()
    tornado.websocket.WebSocketClosedError
[E 2018-01-07 07:08:15.261 SingleUserNotebookApp ioloop:638] Exception in callback <bound method WebSocketMixin.send_ping of ZMQChannelsHandler(bfce21c9-5094-446d-8213-662b31164575)>
    Traceback (most recent call last):
      File "/srv/paws/lib/python3.6/site-packages/tornado/ioloop.py", line 1026, in _run
        return self.callback()
      File "/srv/paws/lib/python3.6/site-packages/notebook/base/zmqhandlers.py", line 197, in send_ping
        self.ping(b'')
      File "/srv/paws/lib/python3.6/site-packages/tornado/websocket.py", line 314, in ping
        raise WebSocketClosedError()
    tornado.websocket.WebSocketClosedError
[E 2018-01-07 07:08:25.568 SingleUserNotebookApp ioloop:638] Exception in callback <bound method WebSocketMixin.send_ping of ZMQChannelsHandler(2d4fda05-9deb-4d55-95fa-74d116f419b8)>
    Traceback (most recent call last):
      File "/srv/paws/lib/python3.6/site-packages/tornado/ioloop.py", line 1026, in _run
        return self.callback()
      File "/srv/paws/lib/python3.6/site-packages/notebook/base/zmqhandlers.py", line 197, in send_ping
        self.ping(b'')
      File "/srv/paws/lib/python3.6/site-packages/tornado/websocket.py", line 314, in ping
        raise WebSocketClosedError()
    tornado.websocket.WebSocketClosedError
[W 2018-01-07 07:30:58.919 SingleUserNotebookApp handlers:257] Replacing stale connection: bfce21c9-5094-446d-8213-662b31164575:9E9E5B7E8A1542A0829B98393D5EC68E
[W 2018-01-07 07:30:58.930 SingleUserNotebookApp handlers:257] Replacing stale connection: 165e4119-78cc-4d1c-83b4-0b6fb23a12ac:8081963C394947C282587F3E3059CC49
[W 2018-01-07 07:30:58.938 SingleUserNotebookApp handlers:257] Replacing stale connection: 2d4fda05-9deb-4d55-95fa-74d116f419b8:0A1BBD0DDBEE45E9957C476D51D8F3B7
[W 2018-01-07 07:30:58.942 SingleUserNotebookApp handlers:257] Replacing stale connection: bfce21c9-5094-446d-8213-662b31164575:9E9E5B7E8A1542A0829B98393D5EC68E
[I 2018-01-07 07:30:58.946 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 598ee897-cb1b-49b6-8e17-596dd14f1d59
[I 2018-01-07 07:30:58.946 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 598ee897-cb1b-49b6-8e17-596dd14f1d59
[I 2018-01-07 07:30:58.946 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 598ee897-cb1b-49b6-8e17-596dd14f1d59
[I 2018-01-07 07:30:58.951 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/598ee897-cb1b-49b6-8e17-596dd14f1d59/channels?session_id=67C32F4F44314ABABEA79C556F36CAE5 (discasto@10.244.5.110) 1512996.03ms
[I 2018-01-07 07:30:58.954 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/598ee897-cb1b-49b6-8e17-596dd14f1d59/channels?session_id=67C32F4F44314ABABEA79C556F36CAE5 (discasto@10.244.5.110) 1412129.54ms
[I 2018-01-07 07:30:58.956 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/598ee897-cb1b-49b6-8e17-596dd14f1d59/channels?session_id=67C32F4F44314ABABEA79C556F36CAE5 (discasto@10.244.5.110) 1381799.71ms
[I 2018-01-07 07:30:58.989 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel bfce21c9-5094-446d-8213-662b31164575
[I 2018-01-07 07:30:58.990 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel bfce21c9-5094-446d-8213-662b31164575
[I 2018-01-07 07:30:58.991 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel bfce21c9-5094-446d-8213-662b31164575
[I 2018-01-07 07:30:58.991 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel bfce21c9-5094-446d-8213-662b31164575
[I 2018-01-07 07:30:58.993 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 165e4119-78cc-4d1c-83b4-0b6fb23a12ac
[I 2018-01-07 07:30:58.994 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 165e4119-78cc-4d1c-83b4-0b6fb23a12ac
[I 2018-01-07 07:30:58.994 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 165e4119-78cc-4d1c-83b4-0b6fb23a12ac
[I 2018-01-07 07:30:58.995 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 165e4119-78cc-4d1c-83b4-0b6fb23a12ac
[I 2018-01-07 07:30:58.995 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 2d4fda05-9deb-4d55-95fa-74d116f419b8
[I 2018-01-07 07:30:59.000 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 2d4fda05-9deb-4d55-95fa-74d116f419b8
[I 2018-01-07 07:30:59.001 SingleUserNotebookApp handlers:193] Adapting to protocol v5.1 for kernel 2d4fda05-9deb-4d55-95fa-74d116f419b8
[I 2018-01-07 07:30:59.002 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/bfce21c9-5094-446d-8213-662b31164575/channels?session_id=9E9E5B7E8A1542A0829B98393D5EC68E (discasto@10.244.5.110) 1538191.70ms
[I 2018-01-07 07:30:59.431 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/bfce21c9-5094-446d-8213-662b31164575/channels?session_id=9E9E5B7E8A1542A0829B98393D5EC68E (discasto@10.244.5.110) 1498598.11ms
[I 2018-01-07 07:30:59.519 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/bfce21c9-5094-446d-8213-662b31164575/channels?session_id=9E9E5B7E8A1542A0829B98393D5EC68E (discasto@10.244.5.110) 2577.53ms
[I 2018-01-07 07:30:59.654 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/bfce21c9-5094-446d-8213-662b31164575/channels?session_id=9E9E5B7E8A1542A0829B98393D5EC68E (discasto@10.244.5.110) 715.15ms
[I 2018-01-07 07:30:59.667 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/165e4119-78cc-4d1c-83b4-0b6fb23a12ac/channels?session_id=8081963C394947C282587F3E3059CC49 (discasto@10.244.5.110) 1522015.40ms
[I 2018-01-07 07:30:59.684 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/165e4119-78cc-4d1c-83b4-0b6fb23a12ac/channels?session_id=8081963C394947C282587F3E3059CC49 (discasto@10.244.5.110) 1485151.34ms
[I 2018-01-07 07:30:59.686 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/165e4119-78cc-4d1c-83b4-0b6fb23a12ac/channels?session_id=8081963C394947C282587F3E3059CC49 (discasto@10.244.5.110) 1383136.78ms
[I 2018-01-07 07:30:59.691 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/165e4119-78cc-4d1c-83b4-0b6fb23a12ac/channels?session_id=8081963C394947C282587F3E3059CC49 (discasto@10.244.5.110) 768.70ms
[I 2018-01-07 07:30:59.722 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/2d4fda05-9deb-4d55-95fa-74d116f419b8/channels?session_id=0A1BBD0DDBEE45E9957C476D51D8F3B7 (discasto@10.244.5.110) 1453436.10ms
[I 2018-01-07 07:30:59.725 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/2d4fda05-9deb-4d55-95fa-74d116f419b8/channels?session_id=0A1BBD0DDBEE45E9957C476D51D8F3B7 (discasto@10.244.5.110) 1382573.60ms
[I 2018-01-07 07:30:59.726 SingleUserNotebookApp log:122] 101 GET /paws/user/discasto/api/kernels/2d4fda05-9deb-4d55-95fa-74d116f419b8/channels?session_id=0A1BBD0DDBEE45E9957C476D51D8F3B7 (discasto@10.244.5.110) 794.77ms
[C 2018-01-07 07:31:08.757 SingleUserNotebookApp notebookapp:1222] received signal 15, stopping
[I 2018-01-07 07:31:08.803 SingleUserNotebookApp notebookapp:1308] Shutting down kernels
[I 2018-01-07 07:31:10.667 SingleUserNotebookApp multikernelmanager:138] Kernel shutdown: 2d4fda05-9deb-4d55-95fa-74d116f419b8
[I 2018-01-07 07:31:10.675 SingleUserNotebookApp multikernelmanager:138] Kernel shutdown: 598ee897-cb1b-49b6-8e17-596dd14f1d59
[I 2018-01-07 07:31:10.689 SingleUserNotebookApp multikernelmanager:138] Kernel shutdown: 165e4119-78cc-4d1c-83b4-0b6fb23a12ac
[I 2018-01-07 07:31:10.703 SingleUserNotebookApp multikernelmanager:138] Kernel shutdown: c6d54e75-e96f-47fe-982e-29650cd44b0b
[I 2018-01-07 07:31:10.717 SingleUserNotebookApp multikernelmanager:138] Kernel shutdown: 90918275-7d9c-49ef-96be-f8b187b9213c
[I 2018-01-07 07:31:10.719 SingleUserNotebookApp multikernelmanager:138] Kernel shutdown: 24434e59-ac1d-43e5-ab94-092f10cfc06c
[I 2018-01-07 07:31:10.723 SingleUserNotebookApp multikernelmanager:138] Kernel shutdown: e29538e2-7bf5-4e41-863a-f4046d76c633
[I 2018-01-07 07:31:10.735 SingleUserNotebookApp multikernelmanager:138] Kernel shutdown: bfce21c9-5094-446d-8213-662b31164575

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• madhuvishy	T184535 PAWS pods hung in state "Terminating"
		Resolved		None	T184500 PAWS is down again

Event Timeline

• madhuvishy created this task.Jan 9 2018, 4:47 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 9 2018, 4:47 PM

• madhuvishy triaged this task as High priority.Jan 9 2018, 4:47 PM

• madhuvishy added a subtask: T184500: PAWS is down again.

• madhuvishy mentioned this in T184500: PAWS is down again.

Looks like the flannel pod in tools-paws-worker-1007 was also stuck in a weird state.

I looked at

yuvipanda@tools-paws-master-01:~$ kubectl --namespace=kube-system get pod -o wide | grep Terminating
kube-flannel-ds-0nvwk                          2/2       Terminating   0          97d       10.68.19.193   tools-paws-worker-1007
yuvipanda@tools-paws-master-01:~$ kubectl --namespace=prod get pod -o wide | grep Terminating
jupyter-aftabbot                           1/1       Terminating        0          43d       10.244.6.187   tools-paws-worker-1007
jupyter-akohdonemma                        1/1       Terminating        1          38d       10.244.6.188   tools-paws-worker-1007
jupyter-discasto                           1/1       Terminating        0          46d       10.244.6.185   tools-paws-worker-1007
jupyter-doc-5ftaxon                        1/1       Terminating        1          47d       10.244.6.184   tools-paws-worker-1007
jupyter-mattho69                           1/1       Terminating        0          38d       10.244.6.189   tools-paws-worker-1007

All these pods were running on 1007, so I checked kube-system

yuvipanda@tools-paws-master-01:~$ kubectl --namespace=kube-system get pod -o wide
NAME                                           READY     STATUS        RESTARTS   AGE       IP             NODE
...
kube-flannel-ds-0nvwk                          2/2       Terminating   0          97d       10.68.19.193   tools-paws-worker-1007
...

I tried draining and cordoning 1007, which didn't help:

yuvipanda@tools-paws-master-01:~$ kubectl drain --delete-local-data --force tools-paws-worker-1007
node "tools-paws-worker-1007" already cordoned
error: DaemonSet-managed pods (use --ignore-daemonsets to ignore): kube-flannel-ds-0nvwk, kube-flannel-ds-nqxq5, kube-proxy-w048m, support-nginx-ingress-controller-g43gp

Decided to reboot tools-worker-1007. The flannel pod went to Error state, and I was able to delete it after. This time a new pod came up running, and all the Jupyter notebook stuck pods went away. So I've put 1007 back in the pool.

yuvipanda@tools-paws-master-01:~$ kubectl uncordon tools-paws-worker-1007
node "tools-paws-worker-1007" uncordoned

Things look fine now.

It seems as it's working properly now. Thanks :-)

Discasto mentioned this in T184566: As usual.... PAWS is down.Jan 9 2018, 10:13 PM

Dvorapa closed subtask T184500: PAWS is down again as Resolved.Jan 12 2018, 6:47 PM

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpUYWdzCgogICAgQW5hbHl0aWNzLUthbmJhbiAoSW4gUHJvZ3Jlc3MpCgpTdWJzY3JpYmVycwpBa2xhcHBlciwgSkFsbGVtYW5kb3UKQXNzaWduZWQgVG8KSkFsbGVtYW5kb3UKQXV0aG9yZWQgQnkKSkFsbGVtYW5kb3UsIEZyaSwgSnVuIDE1CkRlc2NyaXB0aW9uCgpPb3ppZSBqb2JzIGhhdmUgYmVlbiBmYWlsaW5nIGF0IGxlYXN0IGEgZmV3IHRpbWVzIGVhY2guIE1vcmUgaW52ZXN0aWdhdGlvbiBuZWVkZWQuCkpBbGxlbWFuZG91IGNyZWF0ZWQgdGhpcyB0YXNrLkZyaSwgSnVuIDE1LCA3OjIxIEFNCkhlcmFsZCBhZGRlZCBhIHN1YnNjcmliZXI6IEFrbGFwcGVyLiC3IFZpZXcgSGVyYWxkIFRyYW5zY3JpcHRGcmksIEp1biAxNSwgNzoyMSBBTQpKQWxsZW1hbmRvdSBjbGFpbWVkIHRoaXMgdGFzay5GcmksIEp1biAxNSwgNzoyMiBBTQpKQWxsZW1hbmRvdSB1cGRhdGVkIHRoZSB0YXNrIGRlc2NyaXB0aW9uLiAoU2hvdyBEZXRhaWxzKQpKQWxsZW1hbmRvdSBhZGRlZCBhIHByb2plY3Q6IEFuYWx5dGljcy1LYW5iYW4uCkpBbGxlbWFuZG91IG1vdmVkIHRoaXMgdGFzayBmcm9tIE5leHQgVXAgdG8gSW4gUHJvZ3Jlc3Mgb24gdGhlIEFuYWx5dGljcy1LYW5iYW4gYm9hcmQuCkNoYW5nZSBTdWJzY3JpYmVycwpDaGFuZ2UgUHJpb3JpdHkKQXNzaWduIC8gQ2xhaW0KTW92ZSBvbiBXb3JrYm9hcmQKQ2hhbmdlIFByb2plY3QgVGFncwpBbmFseXRpY3MtS2FuYmFuCtcKU2VjdXJpdHkK1wpXaWtpbWVkaWEtVkUtQ2FtcGFpZ25zIChTMi0yMDE4KQrXClNjYXAK1wpTY2FwIChTY2FwMy1BZG9wdGlvbi1QaGFzZTIpCtcKQWJ1c2VGaWx0ZXIK1wpEYXRhLXJlbGVhc2UK1wpIYXNodGFncwrXCkxhYnNEQi1BdWRpdG9yCtcKTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2kK1wpMYW5ndWFnZS0yMDE4LUFwci1KdW5lCtcKTGFuZ3VhZ2UtMjAxOC1KYW4tTWFyCtcKSEhWTQrXCkhBV2VsY29tZQrXCkJvbGQKSXRhbGljcwpNb25vc3BhY2VkCkxpbmsKQnVsbGV0ZWQgTGlzdApOdW1iZXJlZCBMaXN0CkNvZGUgQmxvY2sKUXVvdGUKVGFibGUKVXBsb2FkIEZpbGUKTWVtZQpQcmV2aWV3CkhlbHAKRnVsbHNjcmVlbiBNb2RlClBpbiBGb3JtIE9uIFNjcmVlbgoyMzg0ODJuMzc1IGFkZGVkIHByb2plY3RzOiBTZWN1cml0eSwgV2lraW1lZGlhLVZFLUNhbXBhaWducyAoUzItMjAxOCksIFNjYXAgKFNjYXAzLUFkb3B0aW9uLVBoYXNlMiksIEFidXNlRmlsdGVyLCBEYXRhLXJlbGVhc2UsIEhhc2h0YWdzLCBMYWJzREItQXVkaXRvciwgTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2ksIExhbmd1YWdlLTIwMTgtQXByLUp1bmUsIExhbmd1YWdlLTIwMTgtSmFuLU1hciwgSEhWTSwgSEFXZWxjb21lLlBSRVZJRVcKMjM4NDgybjM3NSBtb3ZlZCB0aGlzIHRhc2sgZnJvbSBJbiBQcm9ncmVzcyB0byBJbiBDb2RlIFJldmlldyBvbiB0aGUgQW5hbHl0aWNzLUthbmJhbiBib2FyZC4KMjM4NDgybjM3NSByZW1vdmVkIEpBbGxlbWFuZG91IGFzIHRoZSBhc3NpZ25lZSBvZiB0aGlzIHRhc2suCjIzODQ4Mm4zNzUgdHJpYWdlZCB0aGlzIHRhc2sgYXMgTG93ZXN0IHByaW9yaXR5LgoyMzg0ODJuMzc1IHJlbW92ZWQgc3Vic2NyaWJlcnM6IEFrbGFwcGVyLCBKQWxsZW1hbmRvdS4KQ29udGVudCBsaWNlbnNlZCB1bmRlciBDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uLVNoYXJlQWxpa2UgMy4wIChDQy1CWS1TQSkgdW5sZXNzIG90aGVyd2lzZSBub3RlZDsgY29kZSBsaWNlbnNlZCB1bmRlciBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSAoR1BMKSBvciBvdGhlciBvcGVuIHNvdXJjZSBsaWNlbnNlcy4gQnkgdXNpbmcgdGhpcyBzaXRlLCB5b3UgYWdyZWUgdG8gdGhlIFRlcm1zIG9mIFVzZSwgUHJpdmFjeSBQb2xpY3ksIGFuZCBDb2RlIG9mIENvbmR1Y3QuILcgV2lraW1lZGlhIEZvdW5kYXRpb24gtyBQcml2YWN5IFBvbGljeSC3IENvZGUgb2YgQ29uZHVjdCC3IFRlcm1zIG9mIFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu