Page MenuHomePhabricator

Rust buildservice failed to clone a repository from GitHub
Closed, DuplicatePublicBUG REPORT

Description

Trying to use buildservice to build mixnmatch tool which worked fine before, now:

tools.mix-n-match@tools-sgebastion-10:~/mixnmatch_rs$ ./build.sh
Waiting for the logs... if the build just started this might take a minute
[place-tools] 2024-04-12T13:11:44.932031979Z 2024/04/12 13:11:44 Copied /ko-app/entrypoint to /tekton/bin/entrypoint
[step-init] 2024-04-12T13:11:45.889494996Z 2024/04/12 13:11:45 Setup /step directories
[place-scripts] 2024-04-12T13:11:47.009149505Z 2024/04/12 13:11:47 Decoded script /tekton/scripts/script-0-plf49
[place-scripts] 2024-04-12T13:11:47.050458411Z 2024/04/12 13:11:47 Decoded script /tekton/scripts/script-2-hlzp5
[place-scripts] 2024-04-12T13:11:47.088662726Z 2024/04/12 13:11:47 Decoded script /tekton/scripts/script-3-dn8ns
[place-scripts] 2024-04-12T13:11:47.118138746Z 2024/04/12 13:11:47 Decoded script /tekton/scripts/script-9-6xhdg
[place-scripts] 2024-04-12T13:11:47.150284405Z 2024/04/12 13:11:47 Decoded script /tekton/scripts/script-10-7nlvf
[place-scripts] 2024-04-12T13:11:47.185087878Z 2024/04/12 13:11:47 Decoded script /tekton/scripts/script-11-dvq8m
[place-scripts] 2024-04-12T13:11:47.221513789Z 2024/04/12 13:11:47 Decoded script /tekton/scripts/script-12-mdwvf
[step-clone] 2024-04-12T13:11:54.905584517Z + '[' false '=' true ]
[step-clone] 2024-04-12T13:11:54.905670744Z + '[' false '=' true ]
[step-clone] 2024-04-12T13:11:54.905684576Z + '[' '$(workspaces.ssl-ca-directory.bound)' '=' true ]
[step-clone] 2024-04-12T13:11:54.905697006Z + CHECKOUT_DIR=/workspace/
[step-clone] 2024-04-12T13:11:54.905777519Z + '[' true '=' true ]
[step-clone] 2024-04-12T13:11:54.905941106Z + cleandir
[step-clone] 2024-04-12T13:11:54.905987143Z + '[' -d /workspace/ ]
[step-clone] 2024-04-12T13:11:54.906161983Z + rm -rf '/workspace//*'
[step-clone] 2024-04-12T13:11:54.907991412Z + rm -rf '/workspace//.[!.]*'
[step-clone] 2024-04-12T13:11:54.909729623Z + rm -rf '/workspace//..?*'
[step-clone] 2024-04-12T13:11:54.911920131Z + test -z
[step-clone] 2024-04-12T13:11:54.911958111Z + test -z
[step-clone] 2024-04-12T13:11:54.911970510Z + test -z
[step-clone] 2024-04-12T13:11:54.912013574Z + /ko-app/git-init '-url=https://github.com/magnusmanske/mixnmatch_rs/' '-revision=' '-refspec=' '-path=/workspace/' '-sslVerify=true' '-submodules=true' '-depth=1' '-sparseCheckoutDirectories='
[step-clone] 2024-04-12T13:14:05.478856302Z {"level":"error","ts":1712927645.478245,"caller":"git/git.go:55","msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin --update-head-ok --force ]: exit status 128\nfatal: unable to access 'https://github.com/magnusmanske/mixnmatch_rs/': Failed to connect to github.com port 443 after 130492 ms: Operation timed out\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:55\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:150\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:53\nruntime.main\n\truntime/proc.go:255"}
[step-clone] 2024-04-12T13:14:05.478930873Z {"level":"fatal","ts":1712927645.4784794,"caller":"git-init/main.go:54","msg":"Error fetching git repository: failed to fetch []: exit status 128","stacktrace":"main.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:54\nruntime.main\n\truntime/proc.go:255"}
[step-copy-builder-to-tmp] 2024-04-12T13:11:48.375194930Z 2024/04/12 13:11:48 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-copy-builder-to-tmp] 2024-04-12T13:14:06.435400868Z 2024/04/12 13:14:06 Skipping step because a previous step failed
[step-prepare] 2024-04-12T13:14:06.660098893Z 2024/04/12 13:14:06 Skipping step because a previous step failed
[step-inject-buildpacks] 2024-04-12T13:14:06.921838330Z 2024/04/12 13:14:06 Skipping step because a previous step failed
[step-copy-stack-toml] 2024-04-12T13:11:49.280592402Z 2024/04/12 13:11:49 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-copy-stack-toml] 2024-04-12T13:14:07.339707142Z 2024/04/12 13:14:07 Skipping step because a previous step failed
[step-detect] 2024-04-12T13:14:07.741439571Z 2024/04/12 13:14:07 Skipping step because a previous step failed
[step-analyze] 2024-04-12T13:11:50.072964425Z 2024/04/12 13:11:50 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-analyze] 2024-04-12T13:14:08.131458058Z 2024/04/12 13:14:08 Skipping step because a previous step failed
[step-restore] 2024-04-12T13:11:50.505412496Z 2024/04/12 13:11:50 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-restore] 2024-04-12T13:14:08.566816456Z 2024/04/12 13:14:08 Skipping step because a previous step failed
[step-build] 2024-04-12T13:14:08.984838636Z 2024/04/12 13:14:08 Skipping step because a previous step failed
[step-fix-nested-procfile-launcher] 2024-04-12T13:14:09.258677005Z 2024/04/12 13:14:09 Skipping step because a previous step failed
[step-fix-permissions] 2024-04-12T13:14:09.490917788Z 2024/04/12 13:14:09 Skipping step because a previous step failed
[step-export] 2024-04-12T13:11:51.882946425Z 2024/04/12 13:11:51 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-export] 2024-04-12T13:14:09.946761637Z 2024/04/12 13:14:09 Skipping step because a previous step failed
[step-results] 2024-04-12T13:14:10.198090104Z 2024/04/12 13:14:10 Skipping step because a previous step failed

Event Timeline

This looks like github timing out:

fatal: unable to access 'https://github.com/magnusmanske/mixnmatch_rs/': Failed to connect to github.com port 443 after 130492 ms: Operation timed out

does it happen more than once? (we might be getting rate-limited or similar)

It did happen three times in a row before I reported it, just worked now. Maybe a github problem or a server not seeing internet?

It did pull the images for the container (might have happened before though) and seemed to be able to resolve github.com (the dns is not local afaik) though of course, things might be cached/not all connectivity broken...

Please report if you find any more, I'll try running a quick 'connectivity check' on the workers.

No connectivity issues to github found:

root@cloudcumin1001:~# cumin 'O{project:tools}' 'nc -w 1 -vz github.com 443'
101 hosts will be targeted:
tools-acme-chief-[3-4].tools.eqiad1.wikimedia.cloud,tools-bastion-[12-13].tools.eqiad1.wikimedia.cloud,tools-checker-5.tools.eqiad1.wikimedia.cloud,tools-cumin-1.tools.eqiad1.wikimedia.cloud,tools-db-[1,3].tools.eqiad1.wikimedia.cloud,tools-docker-registry-[7-8].tools.eqiad1.wikimedia.cloud,tools-elastic-[1-3].tools.eqiad1.wikimedia.cloud,tools-harbor-1.tools.eqiad1.wikimedia.cloud,tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud,tools-k8s-control-[7-9].tools.eqiad1.wikimedia.cloud,tools-k8s-etcd-[22-24].tools.eqiad1.wikimedia.cloud,tools-k8s-haproxy-[5-6].tools.eqiad1.wikimedia.cloud,tools-k8s-ingress-[7-9].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-[102-104].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-nfs-[1-56].tools.eqiad1.wikimedia.cloud,tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud,tools-mail-4.tools.eqiad1.wikimedia.cloud,tools-nfs-2.tools.eqiad1.wikimedia.cloud,tools-package-builder-04.tools.eqiad1.wikimedia.cloud,tools-prometheus-[6-7].tools.eqiad1.wikimedia.cloud,tools-proxy-[7-8].tools.eqiad1.wikimedia.cloud,tools-puppetdb-2.tools.eqiad1.wikimedia.cloud,tools-puppetserver-01.tools.eqiad1.wikimedia.cloud,tools-redis-[5-7].tools.eqiad1.wikimedia.cloud,tools-services-05.tools.eqiad1.wikimedia.cloud,tools-sgebastion-10.tools.eqiad1.wikimedia.cloud,tools-static-15.tools.eqiad1.wikimedia.cloud
OK to proceed on 101 hosts? Enter the number of affected hosts to confirm or "q" to quit: 101
===== NODE GROUP =====                                                                                                                                                                         
(5) tools-elastic-[1-3].tools.eqiad1.wikimedia.cloud,tools-services-05.tools.eqiad1.wikimedia.cloud,tools-sgebastion-10.tools.eqiad1.wikimedia.cloud                                           
----- OUTPUT of 'nc -w 1 -vz github.com 443' -----                                                                                                                                             
Connection to github.com 443 port [tcp/https] succeeded!                                                                                                                                       
===== NODE GROUP =====                                                                                                                                                                         
(96) tools-acme-chief-[3-4].tools.eqiad1.wikimedia.cloud,tools-bastion-[12-13].tools.eqiad1.wikimedia.cloud,tools-checker-5.tools.eqiad1.wikimedia.cloud,tools-cumin-1.tools.eqiad1.wikimedia.cloud,tools-db-[1,3].tools.eqiad1.wikimedia.cloud,tools-docker-registry-[7-8].tools.eqiad1.wikimedia.cloud,tools-harbor-1.tools.eqiad1.wikimedia.cloud,tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud,tools-k8s-control-[7-9].tools.eqiad1.wikimedia.cloud,tools-k8s-etcd-[22-24].tools.eqiad1.wikimedia.cloud,tools-k8s-haproxy-[5-6].tools.eqiad1.wikimedia.cloud,tools-k8s-ingress-[7-9].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-[102-104].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-nfs-[1-56].tools.eqiad1.wikimedia.cloud,tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud,tools-mail-4.tools.eqiad1.wikimedia.cloud,tools-nfs-2.tools.eqiad1.wikimedia.cloud,tools-package-builder-04.tools.eqiad1.wikimedia.cloud,tools-prometheus-[6-7].tools.eqiad1.wikimedia.cloud,tools-proxy-[7-8].tools.eqiad1.wikimedia.cloud,tools-puppetdb-2.tools.eqiad1.wikimedia.cloud,tools-puppetserver-01.tools.eqiad1.wikimedia.cloud,tools-redis-[5-7].tools.eqiad1.wikimedia.cloud,tools-static-15.tools.eqiad1.wikimedia.cloud
----- OUTPUT of 'nc -w 1 -vz github.com 443' -----                                                                                                                                             
Connection to github.com (140.82.114.4) 443 port [tcp/https] succeeded!                                                                                                                        
================                                                                                                                                                                               
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (101/101) [00:05<00:00, 18.65hosts/s]
FAIL |                                                                                                                                                       |   0% (0/101) [00:05<?, ?hosts/s]
100.0% (101/101) success ratio (>= 100.0% threshold) for command: 'nc -w 1 -vz github.com 443'.
100.0% (101/101) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
taavi renamed this task from Rust buildservice fails to Rust buildservice failed to clone a repository from GitHub.Apr 12 2024, 2:45 PM
taavi edited projects, added Toolforge; removed Toolforge Build Service.

Yep, I have seen connectivity issues towards github in the last few days generally on cloud VMs (toolforge included).