Steps to replicate the issue (include links if applicable):
- connect a GitLab project to a kubernetes cluster using "Infrastructure > Kubernetes Clusters"
- connect a project level GitLab runner using "Settings > CI/CD > Runners"
- setup a CI job that:
- deploys a pod
- waits for the pod to be ready
- tries to copy a file from the pod with "kubectl cp"
What happens?:
Job fails with "error: Timeout occured"
What should have happened instead?:
Specified file should be copied into the runner.
Software version (skip for WMF-hosted wikis like Wikipedia):
Using gitlab.wikimedia.org
kindrobot@k8s-cp-demo:~$ gitlab-runner --version Version: 15.10.1 Git revision: dcfb4b66 Git branch: 15-10-stable GO version: go1.19.6 Built: 2023-03-29T13:01:22+0000 OS/Arch: linux/amd64
kindrobot@k8s-cp-demo:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"archive", BuildDate:"2021-06-13T07:08:18Z", GoVersion:"go1.15.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3+k3s1", GitCommit:"01ea3ff27be0b04f945179171cec5a8e11a14f7b", GitTreeState:"clean", BuildDate:"2023-03-27T22:23:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}Other information (browser name/version, screenshots, etc.):
I have a small project that reproduces the problem at https://gitlab.wikimedia.org/kindrobot/kubectl-cp-bug-demo which I would be happy to give access to someone working on this task.
Detailed debug output from the kubectl cp command:
$ kubectl cp -n test hello-nginx:/etc/hostname test-artifacts -v9 I0413 18:05:23.528915 47 loader.go:379] Config loaded from file: /builds/kindrobot/kubectl-cp-bug-demo.tmp/KUBECONFIG I0413 18:05:23.529836 47 round_trippers.go:425] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.20.2 (linux/amd64) kubernetes/faecb19" -H "Authorization: Bearer <masked>" 'https://gitlab.wikimedia.org/-/kubernetes-agent/k8s-proxy/api/v1/namespaces/test/pods/hello-nginx' I0413 18:05:23.716297 47 round_trippers.go:445] GET https://gitlab.wikimedia.org/-/kubernetes-agent/k8s-proxy/api/v1/namespaces/test/pods/hello-nginx 200 OK in 186 milliseconds I0413 18:05:23.717021 47 round_trippers.go:451] Response Headers: I0413 18:05:23.718152 47 round_trippers.go:454] Strict-Transport-Security: max-age=63072000 I0413 18:05:23.718215 47 round_trippers.go:454] Referrer-Policy: strict-origin-when-cross-origin I0413 18:05:23.718226 47 round_trippers.go:454] Date: Thu, 13 Apr 2023 18:05:23 GMT I0413 18:05:23.718303 47 round_trippers.go:454] Content-Type: application/json I0413 18:05:23.718319 47 round_trippers.go:454] Audit-Id: 4b4239f3-016a-4bc4-974b-caa3c2f539ff I0413 18:05:23.718329 47 round_trippers.go:454] Cache-Control: no-cache, private I0413 18:05:23.718338 47 round_trippers.go:454] X-Kubernetes-Pf-Prioritylevel-Uid: 165aff16-f14e-4f06-8a52-ccf1dcbcacb8 I0413 18:05:23.718684 47 round_trippers.go:454] Server: nginx I0413 18:05:23.718695 47 round_trippers.go:454] Vary: Accept-Encoding I0413 18:05:23.718775 47 round_trippers.go:454] Via: 2.0 gitlab-agent/v15.8.0/cdcacd91 I0413 18:05:23.718794 47 round_trippers.go:454] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91 I0413 18:05:23.718804 47 round_trippers.go:454] X-Kubernetes-Pf-Flowschema-Uid: c971f94a-2b85-4cf8-aea6-00c87d9f0c05 I0413 18:05:23.719761 47 request.go:1107] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"hello-nginx","namespace":"test","uid":"962c115a-31e4-4dc6-bb02-8248f0fceb7e","resourceVersion":"2110","creationTimestamp":"2023-04-13T18:05:21Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"hello-nginx\",\"namespace\":\"test\"},\"spec\":{\"containers\":[{\"image\":\"nginx\",\"name\":\"nginx\",\"ports\":[{\"containerPort\":80}]}]}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"v1","time":"2023-04-13T18:05:21Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"nginx\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":80,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:protocol":{}}},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2023-04-13T18:05:23Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.42.0.13\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]},"spec":{"volumes":[{"name":"kube-api-access-dtmzw","projected":{"sources":[{"serviceAccountToken":{"expirationSeconds":3607,"path":"token"}},{"configMap":{"name":"kube-root-ca.crt","items":[{"key":"ca.crt","path":"ca.crt"}]}},{"downwardAPI":{"items":[{"path":"namespace","fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}]}}],"defaultMode":420}}],"containers":[{"name":"nginx","image":"nginx","ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"kube-api-access-dtmzw","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"k8s-cp-demo","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-04-13T18:05:21Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-04-13T18:05:23Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-04-13T18:05:23Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2023-04-13T18:05:21Z"}],"hostIP":"172.16.4.152","podIP":"10.42.0.13","podIPs":[{"ip":"10.42.0.13"}],"startTime":"2023-04-13T18:05:21Z","containerStatuses":[{"name":"nginx","state":{"running":{"startedAt":"2023-04-13T18:05:22Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/library/nginx:latest","imageID":"docker.io/library/nginx@sha256:63b44e8ddb83d5dd8020327c1f40436e37a6fffd3ef2498a6204df23be6e7e94","containerID":"containerd://b9c3dbd7bef9ed78b8f9c9afaefa21c22cf51e8ca373a462b2be1429cbecbbc7","started":true}],"qosClass":"BestEffort"}} I0413 18:05:23.738790 47 round_trippers.go:425] curl -k -v -XPOST -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.20.2 (linux/amd64) kubernetes/faecb19" -H "Authorization: Bearer <masked>" 'https://gitlab.wikimedia.org/-/kubernetes-agent/k8s-proxy/api/v1/namespaces/test/pods/hello-nginx/exec?command=tar&command=cf&command=-&command=%2Fetc%2Fhostname&container=nginx&stderr=true&stdout=true' I0413 18:05:23.901062 47 round_trippers.go:445] POST https://gitlab.wikimedia.org/-/kubernetes-agent/k8s-proxy/api/v1/namespaces/test/pods/hello-nginx/exec?command=tar&command=cf&command=-&command=%2Fetc%2Fhostname&container=nginx&stderr=true&stdout=true 101 Switching Protocols in 162 milliseconds I0413 18:05:23.901103 47 round_trippers.go:451] Response Headers: I0413 18:05:23.901144 47 round_trippers.go:454] X-Stream-Protocol-Version: v4.channel.k8s.io I0413 18:05:23.901155 47 round_trippers.go:454] Server: nginx I0413 18:05:23.901165 47 round_trippers.go:454] Date: Thu, 13 Apr 2023 18:05:23 GMT I0413 18:05:23.901174 47 round_trippers.go:454] Connection: upgrade I0413 18:05:23.901183 47 round_trippers.go:454] Upgrade: SPDY/3.1 I0413 18:05:23.901192 47 round_trippers.go:454] Via: 1.1 gitlab-agent/v15.8.0/cdcacd91 I0413 18:05:23.901226 47 round_trippers.go:454] Via: gRPC/1.0 gitlab-kas/v15.8.0/cdcacd91 F0413 18:05:53.904688 47 helpers.go:115] error: Timeout occured
It appears as if the the SPDY request (used by kubectl cp) is not getting proxied. More information might be available in this Gitlab ticket: https://gitlab.com/gitlab-org/gitlab/-/issues/346248#note_970514106