As part of T398367: Provision user in Cloud VPS hosted Kubernetes cluster for use by nodepool we made a command in the https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning project that can generate a kubeconfig file with credentials for connecting to the Kubernetes cluster in the Cloud VPS project. Using that looks something like:
bd808@mbp03:~/projects/wmf/releng/zuul/tofu-provisioning$ make shell 🖥️ Entering Docker container... Identity added: /dev/fd/63 (zuuldevopsbot) /srv/app/bin/start-socks-proxy.sh: connect: Connection refused /srv/app/bin/start-socks-proxy.sh: line 11: /dev/tcp/127.0.0.1/1080: Connection refused Starting SOCKS5 proxy at 127.0.0.1:1080... Warning: Permanently added 'bastion.wmcloud.org' (ED25519) to the list of known hosts. somebody@deployer:/srv/app$ export TF_LOG= somebody@deployer:/srv/app$ bin/kubeconfig.py --proxy-url http://webproxy:8080 nodepool apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM0VENDQWNtZ0F3SUJBZ0lSQUl6bkhTdm5CRTZvazluWWZyYnh6TTR3RFFZSktvWklodmNOQVFFTEJRQXcKR0RFV01CUUdBMVVFQXd3TmVuVjFiQzFyT0hNdGRqRXlPREFlRncweU5UQTNNVFl5TXpNeU1UWmFGdzB6TURBMwpNVFl5TXpNeU1UWmFNQmd4RmpBVUJnTlZCQU1NRFhwMWRXd3Rhemh6TFhZeE1qZ3dnZ0VpTUEwR0NTcUdTSWIzCkRRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3E4WUZjN0taZzloNVp6UDdjaTdYWkVPRDhCNnl2VmtscDB0aXgKODJPMXZ4MkR6ZmhpM3I2K3A4Skd2TTB2UTJBb2lzT0IzODJpYlVlc1Y0SUVXN3BYekV6UzVvSU0xZTRaZHdCYwpneGZ0QmY4bjIvOHZ4a1lkcjFKQ3krc0xQQlFxOGQ2QUR2MUlQVUFTSGtKRGRMZWlFMS8yS1UrWWRpZ2crcVcrCkQ2Q0gveG1iMTFMbnM2TmZycVk0ZGxURFNHdEJ6R2cyNXp3anVVenlid3l6aHJaSFQ3Y2tZUVg3cnYvUmlOazMKbHo1V2RrUmM4bGRkTzlDbXMraFozM01Mb2ZSL2Q5b3lLR1M0MmFOcy83amdCbE1oTVdXbXByWUpjV1NwcWRsNQpBYjhabWdnUlArcERiUVF1MUtJUjdhRUcxdmFDQjE5WEFxOUdRR1hmY3d3Qnc4NDFBZ01CQUFHakpqQWtNQklHCkExVWRFd0VCL3dRSU1BWUJBZjhDQVFBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUEwR0NTcUdTSWIzRFFFQkN3VUEKQTRJQkFRQmQ3d3E4alhmWEEzMXAxU2psM3FIa29LQ2dyZ2dRb2hjbGtncXFZV0h3Nkx0NUFwYVplYzBLeGRqawp6TXJ6ZW4zR2xrSGdZMmZlc0RVbDM3ZnJzd1VHbk5EbnFraUFzcTRFN09rUTlibTRjYXdDd0tTbm8vOEM5UHVVCnp3UFIxREY2TE5jVVB1bUZwZ1FRK29iYW9SbE1PMHU1bExQSlNhdlA0U1R0SkFZTkUvMFNFV0hjejVIYkdrcksKdlpqb0Z0UGtubVVTd3dEVkVjNUg4bzZrR2Q3eThqdFgyQTV3amoxMElTWGRWMjdXTUx6ZTJnWXcvMUJPazhnUQpwSmhYang4eGVnanVpcTY1SFJ5ZlVDNkI4UkdSamh0V1l2VTIyTkl1V2ZqTTRlWEUxZE0wcjZLUmE1RjN5ODVvClJ3WXJrT1V4RU0rQTl3ZWlkRVp5dkJTeExoUGkKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ== proxy-url: http://webproxy:8080 server: https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443 tls-server-name: 127.0.0.1 name: zuul contexts: - context: cluster: zuul user: nodepool name: zuul current-context: zuul kind: Config users: - name: nodepool user: token: [REDACTED] somebody@deployer:/srv/app$
The nodepool container will need this config file to tell it how to connect to the cluster (https://zuul-ci.org/docs/nodepool/latest/kubernetes.html#attr-providers.[kubernetes].context). It would typically be provisioned in either a $HOME/.kube/config file for the nodepool runtime user or put anywhere else and pointed to by a KUBECONFIG envvar.
These are the interesting parts of the file:
- certificate-authority-data: base64 encoded PEM formatted x509 cert chain for TLS with the Kubernetes API. This non-secret data will change each time we build a new Kubernetes cluster. It is unclear at this point how often that will happen vs being able to upgrade things in place.
- server: k8s-api.svc.zuul.eqiad1.wikimedia.cloud is a service name for the HAProxy layer 4 load balancer that provides IPv6 access to the cluster. In the production config we may want to use the direct IPv6 address rather than the service name which will resolve to an IPv6 that prod can talk to and an IPv4 that can only be used from inside of the Cloud VPS environment.
- token: secret base64 encoded JWT bearer token for authenticating to the Kubernetes API. This will change with each cluster rebuild, similar to certificate-authority-data.
- proxy-url: Tell kubectl and related clients to use http://webproxy:8080 as an HTTP proxy when connecting to the Kubernetes cluster.
Everything else in the kubeconfig file is boiler plate that tells kubectl or similar Kubernetes API clients that the "zuul" cluster and "nodepool" user are used together as the "zuul" context and that the "zuul" context is the active context.