Replace the aging Kubernetes cluster used in Toolforge with a modern deployment of Kubernetes using v1.15 or newer. This will involve:
Started the first run to create the backbone of user RBAC, etc. in tools.
So far, I've run into a problem:
root@tools-k8s-control-3:~# kubectl -n maintain-kubeusers exec -it maintain-kubeusers-ops -- /bin/ash /app # source venv/bin/activate (venv) /app # python maintain_kubeusers.py --gentle-mode --once starting a run Homedir already exists for /data/project/mirador Wrote config in /data/project/mirador/.kube/config Provisioned creds for tool mirador Homedir already exists for /data/project/misc2svg Traceback (most recent call last): File "maintain_kubeusers.py", line 1056, in <module> main() File "maintain_kubeusers.py", line 1030, in main api_server, ca_data, args.gentle_mode File "maintain_kubeusers.py", line 866, in write_kubeconfig self.write_config_file(config) File "maintain_kubeusers.py", line 691, in write_config_file f = os.open(path, os.O_CREAT | os.O_WRONLY | os.O_NOFOLLOW) PermissionError: [Errno 1] Operation not permitted: '/data/project/misc2svg/.kube/config'
I didn't encounter that previously in toolsbeta, and notably, this worked for the tool before it. I wonder if there aren't some configs with an immutable bit set.
That is exactly what I just saw:
[bstorm@labstore1004]:~ $ sudo lsattr /srv/tools/shared/tools/project/misc2svg/.kube/config ----i--------e-- /srv/tools/shared/tools/project/misc2svg/.kube/config
😭
Mentioned in SAL (#wikimedia-cloud) [2019-12-17T01:05:34Z] <bstorm_> beginning the first run of the new maintain-kubeusers in gentle-mode -- but it was just killed by some files setting the immutable bit T214513
Mentioned in SAL (#wikimedia-cloud) [2019-12-17T01:25:13Z] <bstorm_> unset the immutable bit from 1704 tool kubeconfigs T214513
Mentioned in SAL (#wikimedia-cloud) [2019-12-17T01:26:01Z] <bstorm_> running the first run of maintain-kubeusers 2.0 for the new cluster T214513 (more successfully this time)
Failed again after over a thousand tools worked on:
Wrote config in /data/project/commons_describer/.kube/config Could not create podsecuritypolicy for <__main__.User object at 0x7f01c0221210> Traceback (most recent call last): File "maintain_kubeusers.py", line 1056, in <module> main() File "maintain_kubeusers.py", line 1032, in main k8s_api.add_user_access(tools[tool_name]) File "maintain_kubeusers.py", line 644, in add_user_access self.generate_psp(user) File "maintain_kubeusers.py", line 497, in generate_psp _ = self.extensions.create_pod_security_policy(policy) File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 756, in create_pod_security_policy (data) = self.create_pod_security_policy_with_http_info(body, **kwargs) File "/app/venv/lib/python3.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 841, in create_pod_security_policy_with_http_info collection_formats=collection_formats) File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api _request_timeout=_request_timeout) File "/app/venv/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 377, in request body=body) File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 266, in POST body=body) File "/app/venv/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request raise ApiException(http_resp=r) kubernetes.client.rest.ApiException: (422) Reason: Unprocessable Entity HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Tue, 17 Dec 2019 02:21:59 GMT', 'Content-Length': '980'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"PodSecurityPolicy.extensions \"tool-commons_describer-psp\" is invalid: metadata.name: Invalid value: \"tool-commons_describer-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","reason":"Invalid","details":{"name":"tool-commons_describer-psp","group":"extensions","kind":"PodSecurityPolicy","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"tool-commons_describer-psp\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","field":"metadata.name"}]},"code":422}
So we apparently have at least one tool with a '_' in the name. That will break this every time. This also would be invalid as a namespace name, so I have no clue how it would work in the old cluster to begin with. It must have been manually hacked in?
Mentioned in SAL (#wikimedia-cloud) [2019-12-17T04:48:40Z] <bstorm_> completed first run of maintain-kubeusers 2 in the new cluster T214513
Change 558523 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: bastion: raise default value for nproc
Change 558523 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: bastion: raise default value for nproc
Mentioned in SAL (#wikimedia-cloud) [2019-12-17T16:53:33Z] <bstorm_> maintain-kubeusers app deployed fully in tools for new kubernetes cluster T214513 T228499
So we apparently have at least one tool with a '_' in the name. That will break this every time. This also would be invalid as a namespace name, so I have no clue how it would work in the old cluster to begin with. It must have been manually hacked in?
lol? How did that happen?
Apparently, I can never do anything large in Toolforge without discovering a fascinating past mistake of my forebears (and/or myself). I hacked around that and am thankful that it is only four tools.
For the opt-in manual migration period, it would be nice if there could be some kind of helper script like the following (untested):
Change 574568 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] release: remove "gentle mode" from deployment
Change 574625 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] refactor: split up the huge file
I just noticed when testing maintain-kubeusers against a 1.17 cluster that we have a deprecated API version for podsecuritypolicies baked into it. That needs to be fixed. It simply will not run on 1.17 this way.
Change 575322 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge-kubernetes: shut down the old maintain-kubeusers
Change 575325 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: remove the ancient version of kubectl
Change 575322 merged by Bstorm:
[operations/puppet@production] toolforge-kubernetes: shut down the old maintain-kubeusers
Change 574568 merged by Bstorm:
[labs/tools/maintain-kubeusers@master] release: remove "gentle mode" from deployment
Change 575325 merged by Bstorm:
[operations/puppet@production] toolforge: remove the ancient version of kubectl
Change 576469 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] toolforge: remove special configuration for kubernetes on proxy servers
Change 576469 merged by Bstorm:
[operations/puppet@production] toolforge: remove special configuration for kubernetes on proxy servers
Change 574625 merged by Bstorm:
[labs/tools/maintain-kubeusers@master] refactor: split up the huge file
Change 577364 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/maintain-kubeusers@master] refactor: split up the huge file
Change 577364 merged by Bstorm:
[labs/tools/maintain-kubeusers@master] refactor: split up the huge file
There are some small bits of this which are currently stalled, but the core work of the project is completed. Lots of folks helped, but I think @Bstorm deserves the credit not only for her technical work but also for the project management role she played.