Page MenuHomePhabricator

Tool user not allowed to read jobs/status in Kubernetes
Closed, ResolvedPublic

Description

For T320140, my tool needs to launch jobs and check whether they’ve completed or not; I’m currently planning to do this by using the Kubernetes API directly. (I was previously told that this was the preferred approach; T356377 now seems to suggest otherwise, but at this point the k8s API also seems more convenient to me, as it lets me configure the working directory and other useful things.) I’ve found that I can’t use the Python kubernetes method read_namespaced_job_status() to read the job status, as it apparently results in a permission error:

>>> k8s_batch.read_namespaced_job_status(name='sleep-4h', namespace='tool-lucaswerkmeister-test')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/project/lucaswerkmeister-test/www/python/venv/lib/python3.9/site-packages/kubernetes/client/api/batch_v1_api.py", line 2815, in read_namespaced_job_status
    return self.read_namespaced_job_status_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/data/project/lucaswerkmeister-test/www/python/venv/lib/python3.9/site-packages/kubernetes/client/api/batch_v1_api.py", line 2902, in read_namespaced_job_status_with_http_info
    return self.api_client.call_api(
  File "/data/project/lucaswerkmeister-test/www/python/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/data/project/lucaswerkmeister-test/www/python/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/data/project/lucaswerkmeister-test/www/python/venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/data/project/lucaswerkmeister-test/www/python/venv/lib/python3.9/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
  File "/data/project/lucaswerkmeister-test/www/python/venv/lib/python3.9/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '755e7638-561c-4817-9bfd-170310edf4b6', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e75fe91c-f60d-4367-8748-7922aa0b939a', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5e6d0ac-fd38-4495-92d8-4f963bf771a3', 'Date': 'Fri, 09 Feb 2024 18:53:42 GMT', 'Content-Length': '349'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch \"sleep-4h\" is forbidden: User \"lucaswerkmeister-test\" cannot get resource \"jobs/status\" in API group \"batch\" in the namespace \"tool-lucaswerkmeister-test\"","reason":"Forbidden","details":{"name":"sleep-4h","group":"batch","kind":"jobs"},"code":403}

Reading the whole job with read_namespaced_job() works, but it sounds like read_namespaced_job_status() would be slightly nicer by transferring less data. It would be nice to add jobs/status to the permitted permissions (@taavi pointed out rbac.yaml.tpl as a relevant file, which already has some other */status permissions).

Details

TitleReferenceAuthorSource BranchDest Branch
deployment: Grant tools RO access to job statusrepos/cloud/toolforge/maintain-kubeusers!11taavimain-Ia9b313b9fb86248b8dcb85e737ac8b1a311034a5main
Customize query in GitLab

Event Timeline

Note that there's no stability or availability assurance for any of the k8s APIs (raw k8s APIs). I understand they are way more powerful than the APIs/abstractions that we do maintain on top of it, but we can´t offer any kind of assurance that your tools will not break, stop working or misbehave at any point (essentially, let there be dragons).

dcaro triaged this task as Low priority.Feb 12 2024, 9:47 AM
dcaro moved this task from Backlog to Workspace for triaging whenever needed on the Toolforge board.
taavi moved this task from In Review to Done on the Toolforge (Toolforge iteration 05) board.