In T393873 we are creating new ganeti VMs in prod for zuul3.
Out of these 6 VMs, 2 will have the role of an "executor".
These VMs will need to be able to talk to cloud VPS VMs.
This ticket is for the necessary firewall and router config to allow that communication between prod and cloud that is blocked by default.
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| proxy: Allow outbound HTTPS connections to port 6443 | BryanDavis | operations/puppet | production | +1 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | bd808 | T394838 allow new zuul executor VMs in prod to talk to cloud VPS | |||
| Open | hashar | T430479 Zuul executor can not reach WMCS OpenStack API | |||
| Open | None | T431003 zuul-client encrypt fails to retrieve public key |
For the record, the executor and nodepool launcher will need access to :443 for the k8s api server that @bd808 is working on currently for workers. Maybe there are affordances within horizon to allow this traffic without needing to fiddle with prod firewalls?
Case 3: generic network access prod --> cloud is the generally accepted solution for north-south traffic originating from the production network and terminating in Cloud VPS. That solution advises using a web proxy to expose an HTTPS service. Accessing the web proxy from inside the production network space can be done via the webproxy.<datacenter>.wmnet proxy servers. A helper service to maintain a pool of active HTTPS sessions could also be created via envoy inside the production network if needed.
As noted in the Wikitech documentation, we can find another solution if somehow the web proxy is not reliable, secure, or performant enough for the use case. I do not initially have any major concerns about these aspects of the Kubernetes command-and-control channel, but I am happy to hear concerns from others.
I ended up with things related to this need documented on other tasks which is not completely ideal. Let's try to catch everyone up on the current state.
I did initially provision a Cloud VPS web proxy to act as the gateway into the Kubernetes cluster (T396936#10926187). @taavi suggested that more direct connection from prod to the instances could be made using IPv6. After some experimentation I found that OpenStack Magnum could not itself bind IPv6 addresses to the cluster (T396936#10936924).
I decided that a reasonable way to work around this would be to provision an HAproxy instance and configure it to act as a layer 4 proxy. That proxy lives at k8s-api.svc.zuul.eqiad1.wikimedia.cloud which is reachable via IPv6 from the general internet:
bd808@mbp03:~$ ping6 -c3 k8s-api.svc.zuul.eqiad1.wikimedia.cloud PING6(56=40+8+8 bytes) [REDACTED] --> 2a02:ec80:a000:1::2e8 16 bytes from 2a02:ec80:a000:1::2e8, icmp_seq=0 hlim=54 time=80.902 ms 16 bytes from 2a02:ec80:a000:1::2e8, icmp_seq=1 hlim=54 time=85.055 ms 16 bytes from 2a02:ec80:a000:1::2e8, icmp_seq=2 hlim=54 time=78.216 ms --- k8s-api.svc.zuul.eqiad1.wikimedia.cloud ping6 statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 78.216/81.391/85.055/2.813 ms bd808@mbp03:~$ curl -6k https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443 { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"", "reason": "Forbidden", "details": {}, "code": 403 }
From inside the production network we can resolve the IPv6 address:
bd808@deploy1003:~$ host k8s-api.svc.zuul.eqiad1.wikimedia.cloud k8s-api.svc.zuul.eqiad1.wikimedia.cloud has address 172.16.19.147 k8s-api.svc.zuul.eqiad1.wikimedia.cloud has IPv6 address 2a02:ec80:a000:1::2e8
Ping and traceroute do not complete from prod to the Cloud VPS address:
bd808@deploy1003:~$ ping6 -c3 k8s-api.svc.zuul.eqiad1.wikimedia.cloud PING k8s-api.svc.zuul.eqiad1.wikimedia.cloud(zuul-haproxy-01.zuul.eqiad1.wikimedia.cloud (2a02:ec80:a000:1::2e8)) 56 data bytes --- k8s-api.svc.zuul.eqiad1.wikimedia.cloud ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2042ms bd808@deploy1003:~$ traceroute6 --max-hops=10 k8s-api.svc.zuul.eqiad1.wikimedia.cloud traceroute to k8s-api.svc.zuul.eqiad1.wikimedia.cloud (2a02:ec80:a000:1::2e8), 10 hops max, 80 byte packets 1 ae2-1018.cr1-eqiad.wikimedia.org (2620:0:861:102:fe00::1) 0.277 ms 0.282 ms 0.333 ms 2 * * * 3 wan.cloudgw.eqiad1.wikimediacloud.org (2a02:ec80:a000:fe03::3) 0.301 ms 0.177 ms 0.234 ms 4 cloudinstances2b-gw.openstack.eqiad1.wikimediacloud.org (2a02:ec80:a000:fe04::2:1) 0.397 ms 0.360 ms 0.353 ms 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
It is not entirely clear to me if this is expected behavior or not, but since pings and traceroutes to public IPv4 addresses in Cloud VPS also fail it seems reasonable to think it is expected.
What I think is unexpected is that I cannot get to the service via the HTTP proxy service:
bd808@deploy1003:~$ curl -6k --proxy http://webproxy.eqiad.wmnet:8080 https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443 curl: (56) Received HTTP code 403 from proxy after CONNECT
I find this to be unexpected because it is an attempt to access an HTTPS service via the outbound HTTPS proxy. Things work as I expect when an IPv4 address from the Cloud VPS public range is targeted:
bd808@deploy1003:~$ curl --proxy http://webproxy.eqiad.wmnet:8080 https://beta-logs.wmcloud.org <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>401 Unauthorized</title> </head><body> <h1>Unauthorized</h1> <p>This server could not verify that you are authorized to access the document requested. Either you supplied the wrong credentials (e.g., bad password), or your browser doesn't understand how to supply the credentials required.</p> </body></html>
This makes me think that the HTTPS proxy doesn't currently support connecting to IPv6 upstreams.
If we need to use IPv4 we might be better off ripping the complexity of the HAProxy instance out and going back to the Cloud VPS front proxy.
@thcipriani pointed out that in my tests from T394838#11052219 my direct access from my laptop also had a 403 response, but with a different payload. This made us wonder if the proxy was just swallowing the body. I decided to test this theory by bringing a credentials file to deploy1003 so I could try things with auth:
bd808@deploy1003:~/projects/zuul$ set_proxy Proxy set bd808@deploy1003:~/projects/zuul$ KUBECONFIG=kubeconfig-admin kubectl -v3 get namespaces I0731 22:51:44.544737 4026058 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443/api?timeout=32s": Forbidden I0731 22:51:44.546048 4026058 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443/api?timeout=32s": Forbidden I0731 22:51:44.546082 4026058 shortcut.go:89] Error loading discovery information: Get "https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443/api?timeout=32s": Forbidden I0731 22:51:44.547392 4026058 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443/api?timeout=32s": Forbidden I0731 22:51:44.548521 4026058 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443/api?timeout=32s": Forbidden I0731 22:51:44.549728 4026058 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443/api?timeout=32s": Forbidden Unable to connect to the server: Forbidden
No joy.
This is an interesting result though:
$ curl -IL -6 --proxy http://webproxy.eqiad.wmnet:8080 https://google.com HTTP/1.1 200 Connection established HTTP/2 301 location: https://www.google.com/ content-type: text/html; charset=UTF-8 content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-ZDxtVPyblUUw4a7iRPGFmg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp date: Thu, 31 Jul 2025 22:58:47 GMT expires: Sat, 30 Aug 2025 22:58:47 GMT cache-control: public, max-age=2592000 server: gws content-length: 220 x-xss-protection: 0 x-frame-options: SAMEORIGIN alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 HTTP/1.1 200 Connection established HTTP/2 200 content-type: text/html; charset=ISO-8859-1 content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-pHQ4qGjZlbMti-tt2KlG4w' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp accept-ch: Sec-CH-Prefers-Color-Scheme p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info." date: Thu, 31 Jul 2025 22:58:47 GMT server: gws x-xss-protection: 0 x-frame-options: SAMEORIGIN expires: Thu, 31 Jul 2025 22:58:47 GMT cache-control: private set-cookie: AEC=AVh_V2gqneBN2SIYPLA2snFJY1RbA_UXZlfMn0c82LCupmTgrWseLdMwzA; expires=Tue, 27-Jan-2026 22:58:47 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax set-cookie: NID=525=PsiYIsnHTyoGwkxOWTY2zV0qUxbizglHr7oJ2Mbkfg3VUo-keEhBU0FjTWOnZhye-9JeFqGFV2d4Yrk_Q4B6y6w-DkXql0oHoiHdlKgbFOUa0Aufk0IXbVBGU2CkJWjFHlzOpf79vG36EQAe3Pmn2jYyd8_wa4g08knVw-t2sqIhU_HPl36dL_SxW4gs2jyL99DAEqFnNFslKBI; expires=Fri, 30-Jan-2026 22:58:47 GMT; path=/; domain=.google.com; HttpOnly alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
That blows my theory that IPv6 is intrinsically the problem.
profile::installserver::proxy::ssl_ports: - 443 - 873 # rsync used by rpki
Port 6443 is the problem. That port isn't in the acl SSL_ports config for the squid service.
Change #1174842 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[operations/puppet@production] proxy: Allow outbound HTTPS connections to port 6443
Change #1174842 merged by Dzahn:
[operations/puppet@production] proxy: Allow outbound HTTPS connections to port 6443
The suggested firewall change to open port 6443 on installservers (proxies) has been deployed.
Now that the squid proxy allows connecting to port 6443 outbound we can connect to the API from the internal network:
bd808@deploy1003:~/projects/zuul$ curl -6k --proxy http://webproxy.eqiad.wmnet:8080 https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443 { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"", "reason": "Forbidden", "details": {}, "code": 403 }
It turns out that kubectl does not use the https_proxy envvar. Instead a proxy-url setting is needed in the clusters record:
apiVersion: v1 clusters: - cluster: certificate-authority-data: # base64 encoded x509 PEM certificate(s) server: https://k8s-api.svc.zuul.eqiad1.wikimedia.cloud:6443 proxy-url: http://webproxy.eqiad.wmnet:8080 tls-server-name: 127.0.0.1 name: zuul contexts: - context: cluster: zuul user: admin name: zuul current-context: zuul kind: Config users: - name: admin user: client-certificate-data: # base64 encoded x509 PEM certificate
bd808@deploy1003:~/projects/zuul$ KUBECONFIG=kubeconfig-admin kubectl get namespaces NAME STATUS AGE bd808 Active 2d16h default Active 17d kube-node-lease Active 17d kube-public Active 17d kube-system Active 17d
bd808 opened https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/47
kubeconfig: Support --proxy-url declaration
bd808 merged https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/47
kubeconfig: Support --proxy-url declaration
I think updating the squid config has resolved this one. When we configure nodepool to talk to the Kubernetes cluster with T400850: Provision kubeconfig file with credentials for the nodepool user the config we use needs to include a proxy-url: http://webproxy:8080 setting to tell things to use the DC local proxy to reach the Kubernetes API.