[18:45] <icinga-wm> PROBLEM - toolschecker: All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 499 bytes in 0.012 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker
https://wikitech.wikimedia.org/wiki/Incident_documentation/20190910-toolforge-kubernetes
Original report from @JeanFred
Trying to restart either of my tools (via webservice restart) results in ConnectionError
Example with rTHER (a PHP app):
tools.heritage@tools-sgebastion-07:~$ webservice restart Traceback (most recent call last): File "/usr/local/bin/webservice", line 180, in <module> if job.get_state() != Backend.STATE_RUNNING: File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 474, in get_state pod = self._find_obj(pykube.Pod, self.webservice_label_selector) File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 272, in _find_obj o for o in objs File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 125, in __iter__ return iter(self.query_cache["objects"]) File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 115, in query_cache cache["response"] = self.execute().json() File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 99, in execute r = self.api.get(**kwargs) File "/usr/lib/python2.7/dist-packages/pykube/http.py", line 125, in get return self.session.get(*args, **self.get_kwargs(**kwargs)) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 501, in get return self.request('GET', url, **kwargs) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 487, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='k8s-master.tools.wmflabs.org', port=6443): Max retries exceeded with url: /api/v1/namespaces/heritage/pods?labelSelector=tools.wmflabs.org%2Fwebservice-version%3D1%2Cname%3Dheritage%2Ctools.wmflabs.org%2Fwebservice%3Dtrue (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f1adccc5c50>: Failed to establish a new connection: [Errno 111] Connection refused',))
and with R1969 (a Python 2.7 app):
tools.wikiloves@tools-sgebastion-07:~$ webservice restart Traceback (most recent call last): File "/usr/local/bin/webservice", line 180, in <module> if job.get_state() != Backend.STATE_RUNNING: File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 474, in get_state pod = self._find_obj(pykube.Pod, self.webservice_label_selector) File "/usr/lib/python2.7/dist-packages/toollabs/webservice/backends/kubernetesbackend.py", line 272, in _find_obj o for o in objs File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 125, in __iter__ return iter(self.query_cache["objects"]) File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 115, in query_cache cache["response"] = self.execute().json() File "/usr/lib/python2.7/dist-packages/pykube/query.py", line 99, in execute r = self.api.get(**kwargs) File "/usr/lib/python2.7/dist-packages/pykube/http.py", line 125, in get return self.session.get(*args, **self.get_kwargs(**kwargs)) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 501, in get return self.request('GET', url, **kwargs) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 487, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='k8s-master.tools.wmflabs.org', port=6443): Max retries exceeded with url: /api/v1/namespaces/wikiloves/pods?labelSelector=tools.wmflabs.org%2Fwebservice-version%3D1%2Cname%3Dwikiloves%2Ctools.wmflabs.org%2Fwebservice%3Dtrue (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fef82277c50>: Failed to establish a new connection: [Errno 111] Connection refused',))