Page MenuHomePhabricator
Create Task
Maniphest T362958

Buildservice "network is unreachable" error
Open, Needs TriagePublicBUG REPORT
Actions

Description

Aaaaaand another buildservice issue:

[step-build] 2024-04-19T08:31:50.875046067Z        Ign:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
[step-build] 2024-04-19T08:31:50.875396973Z        Err:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
[step-build] 2024-04-19T08:31:50.877313171Z          Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.83), connection timed out [IP: 185.125.190.36 80]
[step-build] 2024-04-19T08:31:50.877391651Z        Ign:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
[step-build] 2024-04-19T08:31:50.878011577Z        Err:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages
[step-build] 2024-04-19T08:31:50.879044946Z          Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) [IP: 185.125.190.36 80]
[step-build] 2024-04-19T08:31:50.879150324Z        Err:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
[step-build] 2024-04-19T08:31:50.880016571Z          Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) [IP: 185.125.190.36 80]
[step-build] 2024-04-19T08:31:50.880241069Z        Err:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
[step-build] 2024-04-19T08:31:50.881781552Z          Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) [IP: 185.125.190.36 80]
[step-build] 2024-04-19T08:31:50.891437819Z        Fetched 2197 kB in 2min 33s (14.4 kB/s)
[step-build] 2024-04-19T08:31:50.943543132Z        Reading package lists...
[step-build] 2024-04-19T08:31:50.954220301Z        W: http://apt.postgresql.org/pub/repos/apt/dists/jammy-pgdg/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
[step-build] 2024-04-19T08:31:50.954502021Z        W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-security/main/binary-amd64/Packages  Error reading from server. Remote end closed connection [IP: 185.125.190.36 80]
[step-build] 2024-04-19T08:31:50.956896767Z        W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-amd64/Packages  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.83), connection timed out [IP: 185.125.190.36 80]
[step-build] 2024-04-19T08:31:50.958436740Z        W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-amd64/Packages  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) [IP: 185.125.190.36 80]
[step-build] 2024-04-19T08:31:50.960003717Z        W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/universe/binary-amd64/Packages  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) [IP: 185.125.190.36 80]
[step-build] 2024-04-19T08:31:50.961497329Z        W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/main/binary-amd64/Packages  Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) [IP: 185.125.190.36 80]
[step-build] 2024-04-19T08:31:50.961693060Z        W: Some index files failed to download. They have been ignored, or old ones used instead.
[step-build] 2024-04-19T08:31:50.969831406Z -----> Fetching .debs for php
[step-build] 2024-04-19T08:31:51.103840435Z E: Unable to locate package php
[step-build] 2024-04-19T08:31:51.106131943Z ERROR: failed to build: exit status 100
[step-fix-nested-procfile-launcher] 2024-04-19T08:31:51.140714246Z 2024/04/19 08:31:51 Skipping step because a previous step failed
[step-fix-permissions] 2024-04-19T08:31:51.363188709Z 2024/04/19 08:31:51 Skipping step because a previous step failed
[step-export] 2024-04-19T08:29:04.727559176Z 2024/04/19 08:29:04 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-export] 2024-04-19T08:31:51.822758081Z 2024/04/19 08:31:51 Skipping step because a previous step failed
[step-results] 2024-04-19T08:31:52.043848005Z 2024/04/19 08:31:52 Skipping step because a previous step failed

Event Timeline

Magnus created this task.Apr 19 2024, 8:46 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 19 2024, 8:46 AM
Magnus added a comment.Apr 19 2024, 8:47 AM

FWIW this is the mix-n-match tool, in case it matters.

bd808 edited projects, added Toolforge; removed Toolforge Build Service.Apr 22 2024, 12:08 AM
bd808 subscribed.Apr 22 2024, 12:15 AM

The root problem here seems to be:

[step-build] 2024-04-19T08:31:50.877313171Z          Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable)

For currently unexplained reasons, the DNS resolver used by the builder process resolved archive.ubuntu.com to an IPv6 address. The network in Cloud VPS & Toolforge is IPv4 only, so this in turn led to nothing working.

dcaro subscribed.Apr 24 2024, 10:44 AM

@Magnus Is this happening often?

I just created an MR to show the node the build is running on, so we can debug this better (https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/42).

Looking into the node that was running this one (I think):

root@tools-k8s-control-9:~# kubectl get taskruns -n image-build mix-n-match-buildpacks-pipelinerun-g49c4-build-from-git -o yaml | grep pod
      "step-build" exited with code 51 (image: "tools-harbor.wmcloud.org/toolforge/heroku-builder@sha256:f5fa557197294852ed12781c472dfabc8125933eb722d5cec24577ecf27031cd"); for logs run: kubectl -n image-build logs mix-n-match-buildpacks-pipelinerun-g49c4-build-from-git-pod -c step-build
  podName: mix-n-match-buildpacks-pipelinerun-g49c4-build-from-git-pod

root@tools-k8s-control-9:~# kubectl get pods -n image-build mix-n-match-buildpacks-pipelinerun-g49c4-build-from-git-pod -o yaml | grep worker
  nodeName: tools-k8s-worker-104

I suspect though that it might have been either us getting rate-limited, or a temporary outage on ubuntu repository side. The stack behavior usually is that it tries ip4, and if it fails then ip6, inside a pod on that same node:

#### IP4 works (so we can't check that it would retry on ip6)
I have no name!@wm-lol-f5fb49b54-dhsts:/workspace$ curl -v http://archive.ubuntu.com
*   Trying 185.125.190.36:80...
* Connected to archive.ubuntu.com (185.125.190.36) port 80 (#0)
> GET / HTTP/1.1
> Host: archive.ubuntu.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 24 Apr 2024 10:35:08 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Vary: Accept-Encoding
< Content-Length: 696
< Content-Type: text/html;charset=UTF-8
< 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /</title>
 </head>
 <body>
<h1>Index of /</h1>
  <table>
   <tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th></tr>
   <tr><th colspan="4"><hr></th></tr>
<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="ubuntu/">ubuntu/</a></td><td align="right">2024-04-24 09:48  </td><td align="right">  - </td></tr>
   <tr><th colspan="4"><hr></th></tr>
</table>
<address>Apache/2.4.29 (Ubuntu) Server at archive.ubuntu.com Port 80</address>
</body></html>
* Connection #0 to host archive.ubuntu.com left intact

##### IPv6 fails as expected
I have no name!@wm-lol-f5fb49b54-dhsts:/workspace$ curl -v -6 http://archive.ubuntu.com
*   Trying 2620:2d:4000:1::19:80...
* Immediate connect fail for 2620:2d:4000:1::19: Network is unreachable
*   Trying 2620:2d:4002:1::101:80...
* Immediate connect fail for 2620:2d:4002:1::101: Network is unreachable
*   Trying 2620:2d:4000:1::16:80...
* Immediate connect fail for 2620:2d:4000:1::16: Network is unreachable
*   Trying 2620:2d:4002:1::102:80...
* Immediate connect fail for 2620:2d:4002:1::102: Network is unreachable
*   Trying 2620:2d:4002:1::103:80...
* Immediate connect fail for 2620:2d:4002:1::103: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server

I say because it seems that on the same run and pod it was able to connect to the server, but it just stopped suddenly:

-----> Updating apt caches                                                                                                                                                                                                                                                                                                                                                                    
       Get:1 http://apt.postgresql.org/pub/repos/apt jammy-pgdg InRelease [123 kB]                                                                                                                                                                                                                                                                                                            
       Get:2 http://apt.postgresql.org/pub/repos/apt jammy-pgdg/main amd64 Packages [498 kB]                                                                                                                                                                                                                                                                                                  
       Ign:3 http://archive.ubuntu.com/ubuntu jammy InRelease                                                                                                                                                                                                                                                                                                                                 
       Get:4 http://archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]                                                                                                                                                                                                                                                                                                               
       Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]                                                                                                                                                                                                                                                                                                                
       Get:3 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]                                                                                                                                                                                                                                                                                                                        
       Get:6 http://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1077 kB]                                                                                                                                                                                                                                                                                                
       Get:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1739 kB]                                                                                                                                                                                                                                                                                                    
       Ign:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages                                                                                                                                                                                                                                                                                                              
       Ign:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages                                                                                                                                                                                                                                                                                                               
       Ign:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages                                                                                                                                                                                                                                                                                                           
       Ign:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages                                                                                                                                                                                                                                                                                                                  
       Ign:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages                                                                                                                                                                                                                                                                                                                      
       Ign:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages                                                                                                                                                                                                                                                                                                              
       Ign:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages                                                                                                                                                                                                                                                                                                               
       Ign:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages                                                                                                                                                                                                                                                                                                           
       Ign:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages                                                                                                                                                                                                                                                                                                                  
       Ign:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages                                                                                                                                                                                                                                                                                                                      
       Ign:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages                                                                                                                                                                                                                                                                                                              
       Ign:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages                                                                                                                                                                                                                                                                                                               
       Ign:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages                                                                                                                                                                                                                                                                                                           
       Ign:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages                                                                                                                                                                                                                                                                                                                  
       Ign:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages                                                                                                                                                                                                                                                                                                                      
       Ign:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages                                                                                                                                                                                                                                                                                                              
       Ign:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages                                                                                                                                                                                                                                                                                                              
       Ign:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages                                                                                                                                                                                                                                                                                                              
       Ign:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages                                                                                                                                                                                                                                                                                                              
       Ign:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages                                                                                                                                                                                                                                                                                                              
       Err:7 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages                                                                                                                                                                                                                                                                                                              
         Error reading from server. Remote end closed connection [IP: 185.125.190.36 80]       ## first time it failed, using ip4                                                                                                                                                                                                                                                                                               
       Ign:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages                                                                                                                                                                                                                                                                                                               
       Ign:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages                                                                                                                                                                                                                                                                                                           
       Ign:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages                                                                                                                                                                                                                                                                                                                  
       Ign:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages                                                                                                                                                                                                                                                                                                               
       Ign:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages                                                                                                                                                                                                                                                                                                                      
       Ign:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages                                                                                                                                                                                                                                                                                                           
       Ign:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages                                                                                                                                                                                                                                                                                                                  
       Ign:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages                                                                                                                                                                                                                                                                                                               
       Ign:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages                                                                                                                                                                                                                                                                                                                      
       Ign:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages                                                                                                                                                                                                                                                                                                           
       Ign:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages                                                                                                                                                                                                                                                                                                                  
       Ign:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages                                                                                                                                                                                                                                                                                                               
       Ign:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages                                                                                                                                                                                                                                                                                                                      
       Ign:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages                                                                                                                                                                                                                                                                                                           
       Ign:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages                                                                                                                                                                                                                                                                                                                  
       Ign:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages                                                                                                                                                                                                                                                                                                               
       Ign:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages                                                                                                                       
       Ign:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages                                                                                                                                                                                                                                                                                                           
       Ign:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages                                                                                                                                                                                                                                                                                                                  
       Err:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages                                                                                                                
         Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.83), connection timed out [IP: 185.125.190.36 80]  # got a timeout on ip4, then network unreachable on ip6
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits