```lang=irc
[18:45] <icinga-wm> PROBLEM - toolschecker: All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 499 bytes in 0.012 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker
```
---
Original report from @JeanFred
Trying to restart either of my tools (via `webservice restart`) results in `ConnectionError`
Example with rTHER (a PHP app):
```lines=10
tools.heritage@tools-sgebastion-07:~$ webservice restart
Traceback (most recent call last):
File "/usr/local/bin/webservice", line 180, in <module>
if job.get_state() != Backend.STATE_RUNNING:
File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 474, in get_state
pod = self._find_obj(pykube.Pod, self.webservice_label_selector)
File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 272, in _find_obj
o for o in objs
File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 125, in __iter__
return iter(self.query_cache["objects"])
File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 115, in query_cache
cache["response"] = self.execute().json()
File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 99, in execute
r = self.api.get(**kwargs)
File "/usr/lib/python2.7/dist-packages/pykube/http.py", line 125, in get
return self.session.get(*args, **self.get_kwargs(**kwargs))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 501, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 487, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='k8s-master.tools.wmflabs.org', port=6443): Max retries exceeded with url: /api/v1/namespaces/heritage/pods?labelSelector=tools.wmflabs.org%2Fwebservice-version%3D1%2Cname%3Dheritage%2Ctools.wmflabs.org%2Fwebservice%3Dtrue (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1adccc5c50>: Failed to establish a new connection: [Errno 111] Connection refused',))
```
and with R1969 (a Python 2.7 app):
```lines=10
tools.wikiloves@tools-sgebastion-07:~$ webservice restart
Traceback (most recent call last):
File "/usr/local/bin/webservice", line 180, in <module>
if job.get_state() != Backend.STATE_RUNNING:
File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 474, in get_state
pod = self._find_obj(pykube.Pod, self.webservice_label_selector)
File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 272, in _find_obj
o for o in objs
File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 125, in __iter__
return iter(self.query_cache["objects"])
File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 115, in query_cache
cache["response"] = self.execute().json()
File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 99, in execute
r = self.api.get(**kwargs)
File "/usr/lib/python2.7/dist-packages/pykube/http.py", line 125, in get
return self.session.get(*args, **self.get_kwargs(**kwargs))
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 501, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 487, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='k8s-master.tools.wmflabs.org', port=6443): Max retries exceeded with url: /api/v1/namespaces/wikiloves/pods?labelSelector=tools.wmflabs.org%2Fwebservice-version%3D1%2Cname%3Dwikiloves%2Ctools.wmflabs.org%2Fwebservice%3Dtrue (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fef82277c50>: Failed to establish a new connection: [Errno 111] Connection refused',))
```