Page MenuHomePhabricator

Change name of commons_describer tool or provide some workaround for Kubernetes/DNS
Closed, ResolvedPublic

Description

Overall, I'm surprised to find a tool that has an underscore in it because that breaks DNS in general.

I am equally curious how this doesn't break the old maintain-kubeusers. As it stands, this will prevent maintain-kubeusers from looping successfully and deployment of the upgraded cluster without either disabling it in kubernetes somehow, deleting the tool (if it is abandoned completely and not in use) or renaming it.

Event Timeline

Bstorm created this task.Dec 17 2019, 2:35 AM
Bstorm moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.
Bstorm triaged this task as High priority.Dec 17 2019, 2:37 AM

Got another one:

Wrote config in /data/project/ru_monuments/.kube/config
Could not create podsecuritypolicy for <__main__.User object at 0x7f667fc12790>
Traceback (most recent call last):
  File "maintain_kubeusers.py", line 1056, in <module>
    main()
  File "maintain_kubeusers.py", line 1032, in main
    k8s_api.add_user_access(tools[tool_name])
  File "maintain_kubeusers.py", line 644, in add_user_access
    self.generate_psp(user)
  File "maintain_kubeusers.py", line 497, in generate_psp
    _ = self.extensions.create_pod_security_policy(policy)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 756, in create_pod_security_policy
    (data) = self.create_pod_security_policy_with_http_info(body, **kwargs)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 841, in create_pod_security_policy_with_http_info
    collection_formats=collection_formats)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 377, in request
    body=body)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 266, in POST
    body=body)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Tue, 17 Dec 2019 03:00:44 GMT', 'Content-Length': '960'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"PodSecurityPolicy.extensions \"tool-ru_monuments-psp\" is invalid: metadata.name: Invalid value: \"tool-ru_monuments-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","reason":"Invalid","details":{"name":"tool-ru_monuments-psp","group":"extensions","kind":"PodSecurityPolicy","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"tool-ru_monuments-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","field":"metadata.name"}]},"code":422}

I admit that I am still surprised.

and another:

Wrote config in /data/project/my_first_tool/.kube/config
Could not create podsecuritypolicy for <__main__.User object at 0x7f51e636c990>
Traceback (most recent call last):
  File "maintain_kubeusers.py", line 1056, in <module>
    main()
  File "maintain_kubeusers.py", line 1032, in main
    k8s_api.add_user_access(tools[tool_name])
  File "maintain_kubeusers.py", line 644, in add_user_access
    self.generate_psp(user)
  File "maintain_kubeusers.py", line 497, in generate_psp
    _ = self.extensions.create_pod_security_policy(policy)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 756, in create_pod_security_policy
    (data) = self.create_pod_security_policy_with_http_info(body, **kwargs)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 841, in create_pod_security_policy_with_http_info
    collection_formats=collection_formats)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 377, in request
    body=body)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 266, in POST
    body=body)
  File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Tue, 17 Dec 2019 03:42:27 GMT', 'Content-Length': '964'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"PodSecurityPolicy.extensions \"tool-my_first_tool-psp\" is invalid: metadata.name: Invalid value: \"tool-my_first_tool-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","reason":"Invalid","details":{"name":"tool-my_first_tool-psp","group":"extensions","kind":"PodSecurityPolicy","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"tool-my_first_tool-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","field":"metadata.name"}]},"code":422}

And one more

Homedir already exists for /data/project/wdq_checker
Wrote config in /data/project/wdq_checker/.kube/config
Could not create podsecuritypolicy for <__main__.User object at 0x7f3f65b13890>
Traceback (most recent call last):
  File "maintain_kubeusers.py", line 1056, in <module>
    main()
  File "maintain_kubeusers.py", line 1032, in main
    k8s_api.add_user_access(tools[tool_name])
  File "maintain_kubeusers.py", line 644, in add_user_access
    self.generate_psp(user)
  File "maintain_kubeusers.py", line 497, in generate_psp
    _ = self.extensions.create_pod_security_policy(policy)

Those appear to be the only four tools with underscores in their names. I eventually hacked some code in that would skip them to finish the initial run.

bd808 added a comment.Dec 17 2019, 5:33 AM

There are a few tools with names that are too long to be used with kubernetes as well (T141100). I think the fix is really just to tell these folks that they need to create new tools to use the Kubernetes cluster. That is functionally the same thing as renaming and avoids the fun of trying to figure out what renaming would require.

I didn't find any that were too long? I completed the whole set.

The name limit in Kubernetes these days is 63 for a namespace (DNS restrictions). Nothing hit that, but maybe there's something in pykube?

We should be good on service name lengths (reading the ticket you referenced): https://github.com/kubernetes/kubernetes/pull/29523
In fact, the old cluster should be if using the right API version (which pykube might not). I might test that. I think these four tools are the only problems in this version. I can just skip them?

Yup, that's old (which is good news!)

tools.bstorm-tool2@tools-sgebastion-07:~$ kubectl apply -f breakingthings.yaml 
service "bstorm-tool2-supercalifragilisticexpialidocious" created

Change 558551 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] hack: work around the underscores in the environment

https://gerrit.wikimedia.org/r/558551

That patch should unblock deployment if we really cannot remove or change these tools (I know we don't have a good story around that).

Change 558551 merged by Bstorm:
[labs/tools/maintain-kubeusers@master] hack: work around the underscores in the environment

https://gerrit.wikimedia.org/r/558551

Bstorm closed this task as Resolved.Dec 19 2019, 9:34 PM
Bstorm claimed this task.

Well, I set up a workaround, so I guess that will have to do. 🙂

Change 577391 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] users: filter out any invalid tool names

https://gerrit.wikimedia.org/r/577391

Change 577391 merged by Bstorm:
[labs/tools/maintain-kubeusers@master] users: filter out any invalid tool names

https://gerrit.wikimedia.org/r/577391