Page MenuHomePhabricator
Create Task
Maniphest T429577

[infra] rate limiting by web.archive.org from within pods
Open, LowPublic
Actions

Description

Toolforge is failing to access through https web.archive.org, tried v4 and v6:

tools.wm-lol@tools-bastion-15:~$ toolforge jobs run --image trixie --command 'curl -4 https://web.archive.org' testcurl                           
tools.wm-lol@tools-bastion-15:~$ cat testcurl*                                                                                                           
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                          
                                 Dload  Upload   Total   Spent    Left  Speed                                                                            
100  145k    0  145k    0     0   355k      0 --:--:-- --:--:-- --:--:--  356k                                                                           
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                          
                                 Dload  Upload   Total   Spent    Left  Speed                                                                            
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0                                                                           
curl: (7) Failed to connect to web.archive.org port 443 after 71 ms: Could not connect to server
tools.wm-lol@tools-bastion-15:~$ toolforge jobs run --command 'curl -6 https://web.archive.org/' testcurl --image trixie
tools.wm-lol@tools-bastion-15:~$ cat testcurl.*
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: web.archive.org

Now it's working for me though, might be a specific worker or a subset, looking.

This was noticed by a user when running the job/bot:

tools.pid-bangladesh-uploadbot2@tools-bastion-15:~$ tail -f ~/pid-bot.out

looking

Workers found with the issue so far:

  • tools-k8s-worker-nfs-80 - fixed itself
  • tools-k8s-worker-nfs-57 - fixed itself
  • tools-k8s-worker-103

Snippet to test in a node:

dcaro@tools-bastion-15:~$ kubectl-sudo run test-pod   --image=docker-registry.svc.toolforge.org/toolforge-trixie-web-sssd:latest   --restart=Never   --overrides='{
    "spec": {
      "nodeName": "tools-k8s-worker-nfs-57"
    }
  }' --rm -it -- /bin/bash
If you don't see a command prompt, try pressing enter.
root@test-pod:/# curl https://web.archive.org
curl: (7) Failed to connect to web.archive.org port 443 after 74 ms: Could not connect to server

Event Timeline

dcaro created this task.Jun 18 2026, 9:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 18 2026, 9:25 AM
R1F4T subscribed.Jun 18 2026, 9:26 AM
TausheefHassan subscribed.Jun 18 2026, 9:27 AM
dcaro added a comment.Jun 18 2026, 9:31 AM

Fonud a node with issues (I think), looking:

tools.wm-lol@tools-bastion-15:~$ kubectl get pods -l 'app.kubernetes.io/name=testcurl' -o json | grep worker
                "nodeName": "tools-k8s-worker-nfs-57",
tools.wm-lol@tools-bastion-15:~$ cat testcurl.*
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to web.archive.org port 443 after 70 ms: Could not connect to server
dcaro added a comment.Jun 18 2026, 9:34 AM

I can curl directly from the worker:

root@tools-k8s-worker-nfs-57:~# curl https://web.archive.org
dcaro updated the task description. (Show Details)Jun 18 2026, 9:36 AM
dcaro updated the task description. (Show Details)Jun 18 2026, 9:41 AM
dcaro added a comment.Jun 18 2026, 9:49 AM

Other external websites seem to work, maybe we got blocked? Oh, now it also works for web archive it seems

dcaro added a comment.Jun 18 2026, 9:53 AM

Worker 80 was failing to curl too, then right after curling google.com, it started working again:

dcaro@tools-bastion-15:~$ kubectl-sudo run test-pod   --image=docker-registry.svc.toolforge.org/toolforge-trixie-web-sssd:latest   --restart=Never   --ov
errides='{                                                                                                                                               
    "spec": {                                                                                                                                            
      "nodeName": "tools-k8s-worker-nfs-80"                                                                                                              
   }                                                                                                                                                     
  }' --rm -it -- curl -v https://web.archive.org -o /dev/null                                                                                            
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                          
                                 Dload  Upload   Total   Spent    Left  Speed                                                                            
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host web.archive.org:443 was resolved.                                   
* IPv6: (none)                                                                                                                                           
* IPv4: 207.241.237.3                                                                                                                                    
*   Trying 207.241.237.3:443...                                                                                                                          
* connect to 207.241.237.3 port 443 from 192.168.18.196 port 45248 failed: Connection refused                                                            
* Failed to connect to web.archive.org port 443 after 76 ms: Could not connect to server                                                                 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0                                                                           
* closing connection #0                                                                                                                                  
curl: (7) Failed to connect to web.archive.org port 443 after 76 ms: Could not connect to server                                                         
pod "test-pod" deleted                                                                                                                                   
pod maintain-kubeusers/test-pod terminated (Error)
dcaro@tools-bastion-15:~$ kubectl-sudo run test-pod   --image=docker-registry.svc.toolforge.org/toolforge-trixie-web-sssd:latest   --restart=Never   --ov
errides='{                                                                                                                                               
    "spec": {                                                                                                                                            
      "nodeName": "tools-k8s-worker-nfs-80"                                                                                                              
   }                                                                                                                                                     
  }' --rm -it -- curl -v https://google.com -o /dev/null                                                                                                 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                          
                                 Dload  Upload   Total   Spent    Left  Speed                                                                            
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host google.com:443 was resolved.
* IPv6: 2607:f8b0:4004:c1f::8b, 2607:f8b0:4004:c1f::8a, 2607:f8b0:4004:c1f::66, 2607:f8b0:4004:c1f::64
* IPv4: 142.251.179.102, 142.251.179.101, 142.251.179.113, 142.251.179.139, 142.251.179.138, 142.251.179.100
*   Trying [2607:f8b0:4004:c1f::8b]:443...
* Immediate connect fail for 2607:f8b0:4004:c1f::8b: Network is unreachable
*   Trying [2607:f8b0:4004:c1f::8a]:443...
* Immediate connect fail for 2607:f8b0:4004:c1f::8a: Network is unreachable
*   Trying [2607:f8b0:4004:c1f::66]:443...
* Immediate connect fail for 2607:f8b0:4004:c1f::66: Network is unreachable
*   Trying [2607:f8b0:4004:c1f::64]:443...
* Immediate connect fail for 2607:f8b0:4004:c1f::64: Network is unreachable
*   Trying 142.251.179.102:443...
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [1565 bytes data]

...
* Connected to google.com (142.251.179.102) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://google.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: google.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.14.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET / HTTP/2
> Host: google.com
> User-Agent: curl/8.14.1
> Accept: */*
> 
* Request completely sent off
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [283 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [283 bytes data]
< HTTP/2 301 
< location: https://www.google.com/
< content-type: text/html; charset=UTF-8
< content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-s6ccIO4YsK8TM5KGCter5A' 'strict-dynamic' 'report-sample' 'unsa
fe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< date: Thu, 18 Jun 2026 09:50:05 GMT
< expires: Sat, 18 Jul 2026 09:50:05 GMT
< cache-control: public, max-age=2592000
< server: gws
< content-length: 220
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<

And from then on:

dcaro@tools-bastion-15:~$ kubectl-sudo run test-pod   --image=docker-registry.svc.toolforge.org/toolforge-trixie-web-sssd:latest   --restart=Never   --ov
errides='{                                                                                                                                               
    "spec": {                                                                                                                                            
      "nodeName": "tools-k8s-worker-nfs-80"                                                                                                              
   }                                                                                                                                                     
  }' --rm -it -- curl -v https://web.archive.org -o /dev/null                                                                                            
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                          
                                 Dload  Upload   Total   Spent    Left  Speed                                                                            
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host web.archive.org:443 was resolved.                                   
* IPv6: (none)                                                                                                                                           
* IPv4: 207.241.237.3                                                                                                                                    
*   Trying 207.241.237.3:443...                                                                                                                          
* ALPN: curl offers h2,http/1.1                                                                                                                          
} [5 bytes data]                                           
...
* Connected to web.archive.org (207.241.237.3) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://web.archive.org/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: web.archive.org]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.14.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET / HTTP/2
> Host: web.archive.org
> User-Agent: curl/8.14.1
> Accept: */*
> 
* Request completely sent off
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
< HTTP/2 200 
< server: nginx
< date: Thu, 18 Jun 2026 09:50:57 GMT
< content-type: text/html
< x-app-server: wwwb-app245-dc8
< x-ts: 200
< x-tr: 3
< server-timing: TR;dur=0,Tw;dur=0,Tc;dur=1
< x-location: Slash
< x-rl: 0
< x-na: 0
< x-page-cache: HIT
< server-timing: HIT
< x-nid: Wikipedia
< referrer-policy: no-referrer-when-downgrade
< permissions-policy: interest-cohort=()
<
dcaro updated the task description. (Show Details)Jun 18 2026, 11:55 AM
dcaro added a subscriber: fgiunchedi.Jun 18 2026, 12:23 PM

@fgiunchedi pointed to check the ips that we are using to get out:

  • bastions use ip6 by default, with the ip4 address being 185.15.56.99
dcaro@tools-bastion-15:~$ curl ifconfig.me
2a02:ec80:a000:1::1c5

dcaro@tools-bastion-15:~$ curl -4 ifconfig.me
185.15.56.99
  • pods use ip4 only, with the ip4 address being 185.15.56.1
root@test2-pod:/# curl ifconfig.me
185.15.56.1

So might be some throttling going on on that ip4 ip

dcaro added a comment.Jun 18 2026, 12:23 PM

The workers themselves use ip4 and that same ip too, let me try to confirm if they get any errors:

root@tools-k8s-control-9:~# curl ifconfig.me
185.15.56.1
dcaro added a comment.Jun 18 2026, 12:24 PM

Yep:

root@tools-k8s-control-9:~# curl -v https://web.archive.org/ -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 207.241.237.3:443...
* connect to 207.241.237.3 port 443 failed: Connection refused
* Failed to connect to web.archive.org port 443 after 128 ms: Couldn't connect to server
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (7) Failed to connect to web.archive.org port 443 after 128 ms: Couldn't connect to server
dcaro triaged this task as Low priority.Jun 18 2026, 3:08 PM
aputhin renamed this task from [infa] unable to access web.archive.org from within pods to [infra] unable to access web.archive.org from within pods.Jun 18 2026, 4:02 PM
aputhin edited projects, added: Toolforge; removed: tools-platform-team.Jun 18 2026, 4:07 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptJun 18 2026, 4:07 PM
dcaro renamed this task from [infra] unable to access web.archive.org from within pods to [infra] rate limiting by web.archive.org from within pods.Jun 18 2026, 5:37 PM
R1F4T added a comment.Jun 18 2026, 8:24 PM

Seems rate limit is the main culprit. now i used a 15s delay between each request and it seems to work but still 2 failed attempt out of 56 not bad.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits