Page MenuHomePhabricator
Create Task
Maniphest T440062

MW Tarball download issues from Europe?
Closed, ResolvedPublic
Actions

Description

Not sure if this is actually an ISP issue on my end, or some connectivity issue through/via ESAMS...

But I was struggling with T439917: makerelease2.py failing to download tarball from releases.wm.o yesterday, and today wget-ing is doing something similar, slow to download and then getting connection closed.

https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz

Doing it from a different host in the UK (different ISP etc), it's also pretty slow, gets connection closed, and has to retry, which makes it seem like a maybe wider spread issue...

It's only a ~50M file, so it's hardly massive.

root@rpi-bookworm-arm64:~# wget https://releases.wikimedia.org/mediawiki/1.43/me                                                                                                             diawiki-core-1.43.10.tar.gz
--2026-10-02 21:35:29--  https://releases.wikimedia.org/mediawiki/1.43/mediawiki                                                                                                             -core-1.43.10.tar.gz
Resolving releases.wikimedia.org (releases.wikimedia.org)... 2a02:ec80:300:ed1a:                                                                                                             :1, 185.15.59.224
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a                                                                                                             ::1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 56470012 (54M) [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    48%[===============================================>                                                     ]  25.98M   273KB/s    in 71s

2026-10-02 21:36:41 (375 KB/s) - Connection closed at byte 27237615. Retrying.

--2026-10-02 21:36:42--  (try: 2)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 29232397 (28M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    50%[++++++++++++++++++++++++++++++++++++++++++++++++=>                                                   ]  27.09M   313KB/s    in 69s

2026-10-02 21:37:52 (16.4 KB/s) - Connection closed at byte 28404463. Retrying.

--2026-10-02 21:37:54--  (try: 3)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 28065549 (27M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    50%[++++++++++++++++++++++++++++++++++++++++++++++++++                                                   ]  27.09M  --.-KB/s    in 68s

2026-10-02 21:39:03 (0.00 B/s) - Connection closed at byte 28404463. Retrying.

--2026-10-02 21:39:06--  (try: 4)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 28065549 (27M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    57%[++++++++++++++++++++++++++++++++++++++++++++++++++=======>                                           ]  31.08M   276KB/s    in 77s

2026-10-02 21:40:24 (52.9 KB/s) - Connection closed at byte 32595183. Retrying.

--2026-10-02 21:40:28--  (try: 5)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 23874829 (23M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    57%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                           ]  31.08M  --.-KB/s    in 70s

2026-10-02 21:41:39 (0.00 B/s) - Connection closed at byte 32595183. Retrying.

--2026-10-02 21:41:45--  (try: 6)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 23874829 (23M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    57%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                           ]  31.08M  --.-KB/s    in 78s

2026-10-02 21:43:03 (0.00 B/s) - Connection closed at byte 32595183. Retrying.

--2026-10-02 21:43:09--  (try: 7)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 23874829 (23M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    57%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                           ]  31.08M  --.-KB/s    in 70s

2026-10-02 21:44:20 (0.00 B/s) - Connection closed at byte 32595183. Retrying.

--2026-10-02 21:44:27--  (try: 8)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 23874829 (23M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    57%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                           ]  31.08M  --.-KB/s    in 72s

2026-10-02 21:45:41 (0.00 B/s) - Connection closed at byte 32595183. Retrying.

--2026-10-02 21:45:49--  (try: 9)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 23874829 (23M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    57%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                           ]  31.08M  --.-KB/s    in 68s

2026-10-02 21:46:59 (0.00 B/s) - Connection closed at byte 32595183. Retrying.

--2026-10-02 21:47:08--  (try:10)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 23874829 (23M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    59%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=>                                         ]  32.05M   284KB/s    in 81s

2026-10-02 21:48:30 (12.2 KB/s) - Connection closed at byte 33609199. Retrying.

--2026-10-02 21:48:40--  (try:11)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 22860813 (22M) remaining [application/x-gzip]
Saving to: 'mediawiki-core-1.43.10.tar.gz'

mediawiki-core-1.43.10.tar.gz                    59%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                         ]  32.05M  --.-KB/s

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
releases: set envoy proxy upstream response timeout to 300.0Dzahnoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Reedy created this task.Fri, Oct 2, 9:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Oct 2, 9:36 PM
Reedy added a subtask: T439917: makerelease2.py failing to download tarball from releases.wm.o.Fri, Oct 2, 9:36 PM
Reedy renamed this task from MW Tarball download issues from Europe? to MW Tarball download issues.Fri, Oct 2, 9:43 PM
Reedy triaged this task as High priority.
Reedy added a project: SRE.
Reedy updated the task description. (Show Details)
Reedy updated the task description. (Show Details)
Reedy updated the task description. (Show Details)
taavi subscribed.Fri, Oct 2, 9:47 PM

I can reproduce when hitting both esams and drmrs, but not when hitting eqiad. In case traceroutes are helpful, see P96552.

Reedy renamed this task from MW Tarball download issues to MW Tarball download issues from Europe?.Fri, Oct 2, 9:48 PM
Reedy updated the task description. (Show Details)
Reedy updated the task description. (Show Details)
taavi added a comment.Fri, Oct 2, 9:53 PM

Can reproduce from the esams bastion directly:

taavi@bast3007 ~ $ wget "https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz"
--2026-10-02 21:50:29--  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Resolving releases.wikimedia.org (releases.wikimedia.org)... 2a02:ec80:300:ed1a::1, 185.15.59.224
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 56470012 (54M) [application/x-gzip]
Saving to: ‘mediawiki-core-1.43.10.tar.gz’

mediawiki-core-1.43.10.tar.gz    54%[=============================>                         ]  29.38M   418KB/s    in 68s     

2026-10-02 21:51:37 (441 KB/s) - Connection closed at byte 30811119. Retrying.

--2026-10-02 21:51:38--  (try: 2)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 25658893 (24M) remaining [application/x-gzip]
Saving to: ‘mediawiki-core-1.43.10.tar.gz’

mediawiki-core-1.43.10.tar.gz    54%[++++++++++++++++++++++++++++++                         ]  29.38M  --.-KB/s    in 70s     

2026-10-02 21:52:50 (0.00 B/s) - Connection closed at byte 30811119. Retrying.

--2026-10-02 21:52:52--  (try: 3)  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 56470012 (54M), 25658893 (24M) remaining [application/x-gzip]
Saving to: ‘mediawiki-core-1.43.10.tar.gz’

mediawiki-core-1.43.10.tar.gz    54%[++++++++++++++++++++++++++++++                         ]  29.38M  --.-KB/s               ^C
• Scott_French subscribed.EditedFri, Oct 2, 10:38 PM

Pulling a couple of data points together, this is reproducible internally when fetching directly from releases.discovery.wmnet (currently a CNAME to releases1003.eqiad.wmnet).

What's puzzling here is that certain network paths - e.g., from hosts in cache sites like esams, drmrs, and ulsfo - to releases1003 result in extremely slow transfer speeds, whereas eqiad (same site) or codfw (~ 32ms RTT) are as fast as one would expect.

In any case, on any network path where the connection is open longer than ~ 1m10s, releases1003 appears to abruptly terminate the connection at that time - e.g., try time curl -v --connect-to releases.wikimedia.org:443:releases.discovery.wmnet:443 https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz -O on an arbitrary host in esams, drmrs, or ulsfo.

I see no evidence of packet loss from these "slow" sites nor unusual network paths (e.g., using mtr).

It feels like something is misconfigured on these newer releases* VMs that results in abnormally slow transfers on higher RTT paths, which is then bumping into a (presumably intentional?) 1m10s timeout.

Edit: I forgot to mention what I mean by "newer" - these (releases1003 and releases2003) appear to have been last reimaged in June, with releases1003 put in service as the backend for the discovery.wmnet CNAME on June 11th (https://gerrit.wikimedia.org/r/1299591).

gerritbot added a comment.Fri, Oct 2, 10:58 PM

Change #1349116 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] releases: set envoy proxy upstream response timeout to 300.0

https://gerrit.wikimedia.org/r/1349116

gerritbot added a project: Patch-For-Review.Fri, Oct 2, 10:59 PM

Change #1349116 merged by Dzahn:

[operations/puppet@production] releases: set envoy proxy upstream response timeout to 300.0

https://gerrit.wikimedia.org/r/1349116

• Scott_French added a subscriber: Dzahn.EditedFri, Oct 2, 11:10 PM

To recap, @Dzahn noticed that envoy had a 65s timeout on the RouteAction associated with the local listener. Indeed, in a scenario where the overall response hits that timeout, after already having sent headers (as here), we expect to see an abrupt connection termination.

With it now bumped out to 300s, I can no longer reproduce the failure from the "slow" sites, though of course the transfer is still slow. We can dig into that another day.

Many thanks for the fix, Daniel!

Reedy mentioned this in T440068: Slow download speed from releases.wikimedia.org.Fri, Oct 2, 11:15 PM
Dzahn added a comment.Fri, Oct 2, 11:15 PM

I set profile::tlsproxy::envoy::upstream_response_timeout: 300 in Hiera on the releases server role with the change linked above.

This reloaded envoy proxy via puppet and looks like it fixed the issue.

The default was 65 from hieradata/common/profile/tlsproxy/envoy.yaml and it was not overridden.

now:

root@releases1003:/home/dzahn# curl -s http://127.0.0.1:9631/config_dump | grep -E '"timeout"'

       "timeout": "300s",
      "timeout": "300s",
"timeout": "300s",
Dzahn added a comment.Fri, Oct 2, 11:16 PM
root@releases1003:/home/dzahn#   curl -s http://127.0.0.1:9631/stats | grep -E 'upstream_rq_timeout|upstream_rq_completed' | grep -v ': 0$'
cluster.local_port_80.external.upstream_rq_completed: 943
cluster.local_port_80.upstream_rq_completed: 943
cluster.local_port_80.upstream_rq_timeout: 3



"completed" grew a lot and no more new timeouts.
Dzahn added a project: Collaboration-Services.Fri, Oct 2, 11:17 PM
Dzahn mentioned this in T439917: makerelease2.py failing to download tarball from releases.wm.o.
Maintenance_bot removed a project: Patch-For-Review.Fri, Oct 2, 11:30 PM
Dzahn lowered the priority of this task from High to Medium.Fri, Oct 2, 11:32 PM
BCornwall moved this task from Backlog to Radar/Not for Service on the Traffic board.Mon, Oct 5, 3:26 PM
LSobanski closed this task as Resolved.Mon, Oct 5, 3:35 PM
LSobanski claimed this task.
LSobanski subscribed.
I believe we can consider this one resolved, with T440068: Slow download speed from releases.wikimedia.org being the follow up task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits