Page MenuHomePhabricator

Provision kubeconfig file with credentials for the nodepool user
Closed, ResolvedPublicFeature

Description

As part of T398367: Provision user in Cloud VPS hosted Kubernetes cluster for use by nodepool we made a command in the https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning project that can generate a kubeconfig file with credentials for connecting to the Kubernetes cluster in the Cloud VPS project. Using that looks something like:

bd808@mbp03:~/projects/wmf/releng/zuul/tofu-provisioning$ make shell
🖥️ Entering Docker container...
Identity added: /dev/fd/63 (zuuldevopsbot)
/srv/app/bin/start-socks-proxy.sh: connect: Connection refused
/srv/app/bin/start-socks-proxy.sh: line 11: /dev/tcp/127.0.0.1/1080: Connection refused
Starting SOCKS5 proxy at 127.0.0.1:1080...
Warning: Permanently added 'bastion.wmcloud.org' (ED25519) to the list of known hosts.
somebody@deployer:/srv/app$ export TF_LOG=
somebody@deployer:/srv/app$ bin/kubeconfig.py --proxy-url http://webproxy:8080 nodepool
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM0VENDQWNtZ0F3SUJBZ0lSQUl6bkhTdm5CRTZvazluWWZyYnh6TTR3RFFZSktvWklodmNOQVFFTEJRQXcKR0RFV01CUUdBMVVFQXd3TmVuVjFiQzFyT0hNdGRqRXlPREFlRncweU5UQTNNVFl5TXpNeU1UWmFGdzB6TURBMwpNVFl5TXpNeU1UWmFNQmd4RmpBVUJnTlZCQU1NRFhwMWRXd3Rhemh6TFhZeE1qZ3dnZ0VpTUEwR0NTcUdTSWIzCkRRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3E4WUZjN0taZzloNVp6UDdjaTdYWkVPRDhCNnl2VmtscDB0aXgKODJPMXZ4MkR6ZmhpM3I2K3A4Skd2TTB2UTJBb2lzT0IzODJpYlVlc1Y0SUVXN3BYekV6UzVvSU0xZTRaZHdCYwpneGZ0QmY4bjIvOHZ4a1lkcjFKQ3krc0xQQlFxOGQ2QUR2MUlQVUFTSGtKRGRMZWlFMS8yS1UrWWRpZ2crcVcrCkQ2Q0gveG1iMTFMbnM2TmZycVk0ZGxURFNHdEJ6R2cyNXp3anVVenlid3l6aHJaSFQ3Y2tZUVg3cnYvUmlOazMKbHo1V2RrUmM4bGRkTzlDbXMraFozM01Mb2ZSL2Q5b3lLR1M0MmFOcy83amdCbE1oTVdXbXByWUpjV1NwcWRsNQpBYjhabWdnUlArcERiUVF1MUtJUjdhRUcxdmFDQjE5WEFxOUdRR1hmY3d3Qnc4NDFBZ01CQUFHakpqQWtNQklHCkExVWRFd0VCL3dRSU1BWUJBZjhDQVFBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUEwR0NTcUdTSWIzRFFFQkN3VUEKQTRJQkFRQmQ3d3E4alhmWEEzMXAxU2psM3FIa29LQ2dyZ2dRb2hjbGtncXFZV0h3Nkx0NUFwYVplYzBLeGRqawp6TXJ6ZW4zR2xrSGdZMmZlc0RVbDM3ZnJzd1VHbk5EbnFraUFzcTRFN09rUTlibTRjYXdDd0tTbm8vOEM5UHVVCnp3UFIxREY2TE5jVVB1bUZwZ1FRK29iYW9SbE1PMHU1bExQSlNhdlA0U1R0SkFZTkUvMFNFV0hjejVIYkdrcksKdlpqb0Z0UGtubVVTd3dEVkVjNUg4bzZrR2Q3eThqdFgyQTV3amoxMElTWGRWMjdXTUx6ZTJnWXcvMUJPazhnUQpwSmhYang4eGVnanVpcTY1SFJ5ZlVDNkI4UkdSamh0V1l2VTIyTkl1V2ZqTTRlWEUxZE0wcjZLUmE1RjN5ODVvClJ3WXJrT1V4RU0rQTl3ZWlkRVp5dkJTeExoUGkKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
    proxy-url: http://webproxy:8080
    server: https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443
    tls-server-name: 127.0.0.1
  name: zuul
contexts:
- context:
    cluster: zuul
    user: nodepool
  name: zuul
current-context: zuul
kind: Config
users:
- name: nodepool
  user:
    token: [REDACTED]

somebody@deployer:/srv/app$

The nodepool container will need this config file to tell it how to connect to the cluster (https://zuul-ci.org/docs/nodepool/latest/kubernetes.html#attr-providers.[kubernetes].context). It would typically be provisioned in either a $HOME/.kube/config file for the nodepool runtime user or put anywhere else and pointed to by a KUBECONFIG envvar.

These are the interesting parts of the file:

  • certificate-authority-data: base64 encoded PEM formatted x509 cert chain for TLS with the Kubernetes API. This non-secret data will change each time we build a new Kubernetes cluster. It is unclear at this point how often that will happen vs being able to upgrade things in place.
  • server: k8s-api.svc.zuul.eqiad1.wikimedia.cloud is a service name for the HAProxy layer 4 load balancer that provides IPv6 access to the cluster. In the production config we may want to use the direct IPv6 address rather than the service name which will resolve to an IPv6 that prod can talk to and an IPv4 that can only be used from inside of the Cloud VPS environment.
  • token: secret base64 encoded JWT bearer token for authenticating to the Kubernetes API. This will change with each cluster rebuild, similar to certificate-authority-data.
  • proxy-url: Tell kubectl and related clients to use http://webproxy:8080 as an HTTP proxy when connecting to the Kubernetes cluster.

Everything else in the kubeconfig file is boiler plate that tells kubectl or similar Kubernetes API clients that the "zuul" cluster and "nodepool" user are used together as the "zuul" context and that the "zuul" context is the active context.

Event Timeline

bd808 changed the subtype of this task from "Task" to "Feature Request".Jul 30 2025, 10:39 PM

@Dzahn I would be glad to talk through any questions you have about this component of the necessary configuration. I added @hashar to the ticket because I think he might be able to help answer some questions too if I'm stumped.

I think if I was coding the puppet for this I would put the three "interesting" config parts into hiera with token in the private.git tree. The file could be generated with a file resource and content => to_yaml(...) or an erb template.

Per findings in T394838: allow new zuul executor VMs in prod to talk to cloud VPS, the clusters[0].cluster object should include proxy-url: http://webproxy:8080.

Change #1179217 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul::main: create notepool sysuser and config template

https://gerrit.wikimedia.org/r/1179217

Dzahn changed the task status from Open to In Progress.Aug 15 2025, 7:45 PM

Change #1179219 had a related patch set uploaded (by Dzahn; author: Dzahn):

[labs/private@master] add fake profile::zuul::main::nodepool::user_token

https://gerrit.wikimedia.org/r/1179219

Change #1179219 merged by Dzahn:

[labs/private@master] add fake profile::zuul::main::nodepool::user_token

https://gerrit.wikimedia.org/r/1179219

Change #1179217 merged by Dzahn:

[operations/puppet@production] zuul::main: create nodepool sysuser and config

https://gerrit.wikimedia.org/r/1179217

@bd808 There is now /etc/nodepool/config on zuul1001/zuul2001 created from an erb template by puppet.

The public data is in hieradata/role/common/zuul/main.yaml and the private string is in private hiera in hieradata/role/common/zuul/main.yaml.

Just need to replace a "snakeoil" string with the actual secret token now.

Just need to replace a "snakeoil" string with the actual secret token now.

Awesome! I just made deploy1003.eqiad.wmnet:/home/bd808/projects/zuul/kubeconfig-nodepool that includes the token data you need in a full working config. Your root superpowers should let you read the file.

Looking at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1179217/5/modules/profile/templates/zuul/nodepool.conf.erb I think there is one bit that got missed; I didn't add the proxy-url: http://webproxy:8080 line from T400850#11058310 to the example in the task description here. You can see it in the kubeconfig-nodepool file. I will update the description here too just for completeness.

Change #1179230 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: add nodepool proxy URL variable

https://gerrit.wikimedia.org/r/1179230

Change #1179230 merged by Dzahn:

[operations/puppet@production] zuul: add nodepool proxy URL variable

https://gerrit.wikimedia.org/r/1179230

Change #1179231 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul::main: add mode 0550 to file holding a secret

https://gerrit.wikimedia.org/r/1179231

Change #1179231 merged by Dzahn:

[operations/puppet@production] zuul::main: add mode 0550 to file holding a secret

https://gerrit.wikimedia.org/r/1179231

@bd808 Done! The proxy-url line has been added and the real token is in the config as well.

[zuul1001:~] $ sudo cat /etc/nodepool/config

optimistically calling it resolved