notes from debugging:

• Apache isn't the problem. It sent the whole 99 MB file in 0.23 seconds (about 435 MB/s), so the disk and Apache are fine. (from curl testing on releases1003 directly to httpd on local).

• Envoy is fast too. about 534 MB/s locally, faster than plain Apache (from curl testing on releases1003 to the envoy port)

Change #1353476 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] service: mark releases as active/passive

https://gerrit.wikimedia.org/r/1353476

Tried downloading mediawiki tarball from my home ISP after noticing slow transfer speed with the API when receiving large responses. releases.wikimedia.org is served by the text addresses, and resolves to the esams address here. However, if I point it to eqiad, codfw or even magru, the speed can goes up to several MB/s (with ulsfo it's mostly the same, while with eqsin it goes below 100kB/s). Not sure if it's just me, or the two European cache DCs have some issues recently.

The issue is the speed at which esams is pulling the file from codfw, which happens every time. Tbh it doesn't make much sense for users to connect to esams for this given that is the case, but that is how the CDN currently works.

cmooney@bast3007:~$ wget https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.9.tar.gz 
--2026-10-09 00:03:33--  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.9.tar.gz
Resolving releases.wikimedia.org (releases.wikimedia.org)... 2a02:ec80:300:ed1a::1, 185.15.59.224
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 93194737 (89M) [application/x-gzip]
Saving to: ‘mediawiki-1.43.9.tar.gz.1’

mediawiki-1.43.9.tar.gz.1   46%[==================>                      ]  41.35M   359KB/s    eta 3m 11s

I get the same at home here. A few hundred KB/sec by default (I hit esams), but if I go direct to codfw I get 30MB/sec.

Looking at the backhaul I'm not sure what the exact issue is. I disabled the current path traffic is taking to use different WAN links across the Atlantic but the speed stayed at a few undred KBps. All those links look clean in terms of usage and packet loss so I'm not sure if it's the backhaul itself.

For your own use it's probably best to always use eqiad or codfw, they will perform better as the file needs to come from the US anyway. As for the general performance issue I'll see if I can find anything else that might explain it.

According to https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals, esams and drmrs should not have much load on the public side right now (it's midnight in Europe and the region served by them). Yet the transfer speed is still slow.

more notes (that I did not get to save yet, from earlier)

• ATS isn't the cause, it's just slow without it (from curl testing from cp4041 to releases1003)
• slowdown is in the ulsfo → releases1003 TCP connection itself (from debugging with "ss")

Mentioned in SAL (#wikimedia-operations) [2026-10-09T01:14:34Z] <mutante> releases1003 - /usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr T440068 - downloads 120x times faster

• Ganeti host isnt dropping packets either
• roughly 0.8% loss is on the network beyond ganeti1033: the switches, the eqiad↔ulsfo transport, or the cp side
• tested changing TCP congestion control from cubic (default) to BBR. result: download now 120x faster

before:

cp4041:~] $ curl -s -o /dev/null --resolve releases.wikimedia.org:443:10.64.48.34 -w 'code=%{http_code} size=%{size_download} speed=%{speed_download}B/s total=%{time_total}s\n' https://releases.wikimedia.org/mediawiki/1.46/mediawiki-1.46.2.tar.gz


code=200 size=77682599 speed=256162B/s total=303.255222s

on releases1003:

/usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr

after:

[cp4041:~] $ curl -s -o /dev/null --resolve releases.wikimedia.org:443:10.64.48.34 -w 'code=%{http_code} size=%{size_download} speed=%{speed_download}B/s total=%{time_total}s\n' https://releases.wikimedia.org/mediawiki/1.46/mediawiki-1.46.2.tar.gz

code=200 size=99029682 speed=30951553B/s total=3.199506s

How about now, everybody?

Has improved for me. Though I’m curious why those dedicated inter-DC links have a relatively high loss rate.

In T440068#12414454, @Dzahn wrote:

• roughly 0.8% loss is on the network beyond ganeti1033: the switches, the eqiad↔ulsfo transport, or the cp side

I'm not seeing that reflected in the blackbox probe stats, but we can investigate more if it persists, how did you measure this?

• tested changing TCP congestion control from cubic (default) to BBR. result: download now 120x faster

Wow I had considered that as a factor last night. But damn, the difference really is night and day:

cmooney@bast3007:~$ wget https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.9.tar.gz 
--2026-10-09 08:16:00--  https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.9.tar.gz
Resolving releases.wikimedia.org (releases.wikimedia.org)... 2a02:ec80:300:ed1a::1, 185.15.59.224
Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 93194737 (89M) [application/x-gzip]
Saving to: ‘mediawiki-1.43.9.tar.gz.3’

mediawiki-1.43.9.tar.gz.3 100%[=====================================>]  88.88M  24.6MB/s    in 4.7s    

2026-10-09 08:16:05 (18.9 MB/s) - ‘mediawiki-1.43.9.tar.gz.3’ saved [93194737/93194737]

I mean I was a fan of BBR already but an improvement like that is still amazing to see. I see the same improvement from home coming from esams too.

In T440068#12414362, @cmooney wrote:

The issue is the speed at which esams is pulling the file from codfw, which happens every time.

Just checking on this part. This should be about "esams <-> eqiad" link. the currently active backend is releases1003 in eqiad.

how did you measure this?

Got data from the TCP counters on releases1003 to cp4041. ss -tin dst $(dig +short cp4041.ulsfo.wmnet)

And ss -tinH state established '( sport = :443 )' | awk 'NR%2{p=$4;next}{t=s=r="-";for(i=1;i<=NF;i++){if($i~/^rtt:/)t=$i;if($i~/^segs_out:/)s=$i;if($i~/^retrans:/)r=$i}print p,t,s,r}' | sort for all live connections from the caches to envoy.

Then from the counter output AI claimed to have calculated it by "retransmitted segments divided by segments sent, per live connection to a cache".

retrans:0/12 and segs_out:1459, so 12 / 1459 = 0.82%

(cp3070 0.93%, cp1102 0.03%, cp1114 0.13%):

In T440068#12416633, @Dzahn wrote:

Just checking on this part. This should be about "esams <-> eqiad" link. the currently active backend is releases1003 in eqiad.

Ah ok. I hadn't actually checked which core site it was pulling from, I got my wires crossed thought it was codfw. But the statement still applies, the issue was internal between our sites, rather than from esams, ulsfo etc to the users.

how did you measure this?

Got data from the TCP counters on releases1003 to cp4041. ss -tin dst $(dig +short cp4041.ulsfo.wmnet)

And ss -tinH state established '( sport = :443 )' | awk 'NR%2{p=$4;next}{t=s=r="-";for(i=1;i<=NF;i++){if($i~/^rtt:/)t=$i;if($i~/^segs_out:/)s=$i;if($i~/^retrans:/)r=$i}print p,t,s,r}' | sort for all live connections from the caches to envoy.

Then from the counter output AI claimed to have calculated it by "retransmitted segments divided by segments sent, per live connection to a cache".

That logic seems sound yep. I don't see it reflected in say an MTR, but I think if performance is now ok we can move on and not try to rabbit hole on the discrepancy.

cmooney@releases1003:~$ mtr -b -w -c 1000 cp4041.ulsfo.wmnet 
Start: 2026-10-09T15:21:33+0000
HOST: releases1003                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- et-1-0-5-1020.cr2-eqiad.wikimedia.org (2620:0:861:107:fe00::3)      0.0%  1000    1.5   2.6   0.5  81.8   9.7
  2.|-- et-0-0-0-41.cr4-ulsfo.wikimedia.org (2620:0:863:fe0e::2)            0.1%  1000   79.5  74.8  74.3 106.6   2.1
  3.|-- ethernet-1-56-100.asw1-22-ulsfo.wikimedia.org (2620:0:863:fe0a::2)  0.1%  1000   75.0  74.8  74.6  78.7   0.3
  4.|-- cp4041.ulsfo.wmnet (2620:0:863:101:10:128:0:25)                     0.0%  1000   66.1  65.9  65.8  68.7   0.2

Change #1354036 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] releases: use BBR TCP congestion control

https://gerrit.wikimedia.org/r/1354036

Change #1354037 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: use BBR TCP congestion control

https://gerrit.wikimedia.org/r/1354037

Change #1354036 merged by Dzahn:

[operations/puppet@production] releases: use BBR TCP congestion control

https://gerrit.wikimedia.org/r/1354036

Change #1354037 abandoned by Dzahn:

[operations/puppet@production] gerrit: use BBR TCP congestion control

Reason:

tested from esams and ulsfo at about 4–5 MB/s with cubic, limited by archive generation, so not needed.

https://gerrit.wikimedia.org/r/1354037

Follow-up tests, all with cubic (default), 100 MB / 42 MB files:

* people2004 (codfw) -> cp3070 (esams), direct: 24 MB/s
* people1005 (eqiad) -> cp3070 (esams), direct: 45 MB/s
* gerrit (gitiles archive) -> cp3070 direct: 4.3 MB/s, via ulsfo CDN: 4.7 MB/s (limited by archive generation, not network)
* releases1003 -> cp3070/cp4041 before BBR: ~250 KB/s

people1005/2004 and releases1003 are all trixie KVM VMs with the same TCP sysctls, qdisc and offloads, and no conntrack drops.

So it's not trixie, not the eqiad network path and not general cubic behaviour. It looks specific to releases1003 or its Ganeti node (ganeti1033).

BBR for releases is permanent now: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1354036

BBR for gerrit (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1354037) is not needed and will be abandoned.

Next: live-migrate releases1003 to ganeti1038, retest with cubic.