Overall, I'm surprised to find a tool that has an underscore in it because that breaks DNS in general.
I am equally curious how this doesn't break the old maintain-kubeusers. As it stands, this will prevent maintain-kubeusers from looping successfully and deployment of the upgraded cluster without either disabling it in kubernetes somehow, deleting the tool (if it is abandoned completely and not in use) or renaming it.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | • Bstorm | T246122 Upgrade the Toolforge Kubernetes cluster to v1.16 | |||
| Restricted Task | |||||
| Resolved | bd808 | T232536 Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail | |||
| Resolved | • Bstorm | T236565 "tools" Cloud VPS project jessie deprecation | |||
| Resolved | aborrero | T101651 Set up toolsbeta more fully to help make testing easier | |||
| Resolved | • Bstorm | T166949 Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf) | |||
| Resolved | • Bstorm | T246059 Add admin account creation to maintain-kubeusers | |||
| Resolved | • Bstorm | T154504 Make webservice backend default to kubernetes | |||
| Declined | None | T245230 Investigate cpu/ram requests and limits for DaemonSets pods | |||
| Resolved | • Bstorm | T214513 Deploy and migrate tools to a Kubernetes v1.15 or newer cluster | |||
| Resolved | • Bstorm | T240922 Change name of commons_describer tool or provide some workaround for Kubernetes/DNS |
Got another one:
Wrote config in /data/project/ru_monuments/.kube/config
Could not create podsecuritypolicy for <__main__.User object at 0x7f667fc12790>
Traceback (most recent call last):
File "maintain_kubeusers.py", line 1056, in <module>
main()
File "maintain_kubeusers.py", line 1032, in main
k8s_api.add_user_access(tools[tool_name])
File "maintain_kubeusers.py", line 644, in add_user_access
self.generate_psp(user)
File "maintain_kubeusers.py", line 497, in generate_psp
_ = self.extensions.create_pod_security_policy(policy)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 756, in create_pod_security_policy
(data) = self.create_pod_security_policy_with_http_info(body, **kwargs)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 841, in create_pod_security_policy_with_http_info
collection_formats=collection_formats)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
_request_timeout=_request_timeout)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 377, in request
body=body)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 266, in POST
body=body)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Tue, 17 Dec 2019 03:00:44 GMT', 'Content-Length': '960'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"PodSecurityPolicy.extensions \"tool-ru_monuments-psp\" is invalid: metadata.name: Invalid value: \"tool-ru_monuments-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","reason":"Invalid","details":{"name":"tool-ru_monuments-psp","group":"extensions","kind":"PodSecurityPolicy","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"tool-ru_monuments-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","field":"metadata.name"}]},"code":422}I admit that I am still surprised.
and another:
Wrote config in /data/project/my_first_tool/.kube/config
Could not create podsecuritypolicy for <__main__.User object at 0x7f51e636c990>
Traceback (most recent call last):
File "maintain_kubeusers.py", line 1056, in <module>
main()
File "maintain_kubeusers.py", line 1032, in main
k8s_api.add_user_access(tools[tool_name])
File "maintain_kubeusers.py", line 644, in add_user_access
self.generate_psp(user)
File "maintain_kubeusers.py", line 497, in generate_psp
_ = self.extensions.create_pod_security_policy(policy)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 756, in create_pod_security_policy
(data) = self.create_pod_security_policy_with_http_info(body, **kwargs)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 841, in create_pod_security_policy_with_http_info
collection_formats=collection_formats)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
_request_timeout=_request_timeout)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 377, in request
body=body)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 266, in POST
body=body)
File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Tue, 17 Dec 2019 03:42:27 GMT', 'Content-Length': '964'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"PodSecurityPolicy.extensions \"tool-my_first_tool-psp\" is invalid: metadata.name: Invalid value: \"tool-my_first_tool-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","reason":"Invalid","details":{"name":"tool-my_first_tool-psp","group":"extensions","kind":"PodSecurityPolicy","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"tool-my_first_tool-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","field":"metadata.name"}]},"code":422}And one more
Homedir already exists for /data/project/wdq_checker
Wrote config in /data/project/wdq_checker/.kube/config
Could not create podsecuritypolicy for <__main__.User object at 0x7f3f65b13890>
Traceback (most recent call last):
File "maintain_kubeusers.py", line 1056, in <module>
main()
File "maintain_kubeusers.py", line 1032, in main
k8s_api.add_user_access(tools[tool_name])
File "maintain_kubeusers.py", line 644, in add_user_access
self.generate_psp(user)
File "maintain_kubeusers.py", line 497, in generate_psp
_ = self.extensions.create_pod_security_policy(policy)Those appear to be the only four tools with underscores in their names. I eventually hacked some code in that would skip them to finish the initial run.
There are a few tools with names that are too long to be used with kubernetes as well (T141100). I think the fix is really just to tell these folks that they need to create new tools to use the Kubernetes cluster. That is functionally the same thing as renaming and avoids the fun of trying to figure out what renaming would require.
The name limit in Kubernetes these days is 63 for a namespace (DNS restrictions). Nothing hit that, but maybe there's something in pykube?
We should be good on service name lengths (reading the ticket you referenced): https://github.com/kubernetes/kubernetes/pull/29523
In fact, the old cluster should be if using the right API version (which pykube might not). I might test that. I think these four tools are the only problems in this version. I can just skip them?
Yup, that's old (which is good news!)
tools.bstorm-tool2@tools-sgebastion-07:~$ kubectl apply -f breakingthings.yaml service "bstorm-tool2-supercalifragilisticexpialidocious" created
Change 558551 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] hack: work around the underscores in the environment
That patch should unblock deployment if we really cannot remove or change these tools (I know we don't have a good story around that).
Change 558551 merged by Bstorm:
[labs/tools/maintain-kubeusers@master] hack: work around the underscores in the environment
For future reference, there was T176027: Tools with "_" in their name or names longer than 63 characters do not get Kubernetes namespaces created
Change 577391 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] users: filter out any invalid tool names
Change 577391 merged by Bstorm:
[labs/tools/maintain-kubeusers@master] users: filter out any invalid tool names