Maniphest T120425

dumps.wikimedia.org seems to have poor throughput towards some destinations
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	Nemo_bis
	Dec 4 2015, 9:23 PM

Description

Downloading the same file at the same time on two servers goes at 1.8 MB/s on one, 200 KB/s on the other; the latter happily downloads at 3 MB/s from upload.wikimedia.org (via AMS; couldn't find a functioning eqiad URL for that).

$ wget -O /dev/null --no-check-certificate https://upload.wikimedia.org/wikipedia/commons/f/f2/Wikiconference_in_Moscow_day_2_part_5.webm
--2015-12-04 22:16:17--  https://upload.wikimedia.org/wikipedia/commons/f/f2/Wikiconference_in_Moscow_day_2_part_5.webm
Resolving upload.wikimedia.org... 91.198.174.208, 2620:0:862:ed1a::2:b
Connecting to upload.wikimedia.org|91.198.174.208|:443... connected.
WARNING: certificate common name “*.wikipedia.org” doesn’t match requested host name “upload.wikimedia.org”.
HTTP request sent, awaiting response... 200 OK
Length: 3573382210 (3.3G) [video/webm]
Saving to: “/dev/null”

10% [=========>                                                                                            ] 385,027,006 2.47M/s  eta 18m 17s
$ traceroute upload.wikimedia.org
traceroute to upload.wikimedia.org (91.198.174.208), 30 hops max, 60 byte packets
 1  109-204-236-1.kierratyskone.fi (109.204.236.1)  0.880 ms  0.853 ms  1.055 ms
 2  host-109-204-196-145.tp-fne.tampereenpuhelin.net (109.204.196.145)  1.776 ms  2.078 ms  2.056 ms
 3  host-109-204-196-129.tp-fne.tampereenpuhelin.net (109.204.196.129)  4.473 ms  5.434 ms  4.414 ms
 4  ae4-100.RT.HMO.HKI.FI.retn.net (87.245.248.161)  4.656 ms  4.634 ms  4.599 ms
 5  et1010-4.RT.TC2.AMS.NL.retn.net (87.245.232.114)  32.540 ms  32.519 ms  32.242 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
$ traceroute dumps.wikimedia.org
traceroute to dumps.wikimedia.org (208.80.154.11), 30 hops max, 60 byte packets
 1  109-204-236-1.kierratyskone.fi (109.204.236.1)  150.904 ms  150.879 ms  151.070 ms
 2  host-109-204-196-145.tp-fne.tampereenpuhelin.net (109.204.196.145)  1.658 ms  1.888 ms  1.868 ms
 3  host-109-204-196-129.tp-fne.tampereenpuhelin.net (109.204.196.129)  4.332 ms  4.316 ms  4.295 ms
 4  hls-b3-link.telia.net (213.248.92.101)  4.523 ms  4.495 ms  4.477 ms
 5  s-bb3-link.telia.net (213.155.134.106)  11.912 ms s-bb3-link.telia.net (80.91.245.34)  11.900 ms s-bb4-link.telia.net (62.115.142.242)  12.371 ms
 6  kbn-bb3-link.telia.net (62.115.139.167)  21.056 ms kbn-bb4-link.telia.net (62.115.139.177)  18.418 ms kbn-bb3-link.telia.net (62.115.139.167)  21.158 ms
 7  nyk-bb2-link.telia.net (213.155.134.52)  109.327 ms nyk-bb1-link.telia.net (62.115.141.101)  112.355 ms nyk-bb1-link.telia.net (80.91.249.21)  100.568 ms
 8  ash-bb4-link.telia.net (62.115.136.201)  125.541 ms  125.348 ms ash-bb3-link.telia.net (62.115.137.60)  109.542 ms
 9  ash-b1-link.telia.net (62.115.113.211)  112.030 ms  112.640 ms ash-b1-link.telia.net (213.155.130.3)  114.330 ms
10  wikimedia-ic-308845-ash-b2.c.telia.net (80.239.132.226)  117.999 ms  113.624 ms  113.622 ms
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
$ traceroute upload-lb.eqiad.wikimedia.org
traceroute to upload-lb.eqiad.wikimedia.org (208.80.154.240), 30 hops max, 60 byte packets
 1  109-204-236-1.kierratyskone.fi (109.204.236.1)  0.807 ms  0.783 ms  0.763 ms
 2  host-109-204-196-145.tp-fne.tampereenpuhelin.net (109.204.196.145)  1.442 ms  1.678 ms  1.888 ms
 3  host-109-204-196-129.tp-fne.tampereenpuhelin.net (109.204.196.129)  4.374 ms  4.358 ms  4.336 ms
 4  hls-b3-link.telia.net (213.248.92.101)  4.552 ms  4.532 ms  4.503 ms
 5  s-bb3-link.telia.net (62.115.113.106)  11.951 ms s-bb3-link.telia.net (62.115.113.104)  11.937 ms s-bb3-link.telia.net (62.115.114.44)  11.166 ms
 6  kbn-bb3-link.telia.net (62.115.139.171)  21.132 ms kbn-bb3-link.telia.net (62.115.139.60)  20.204 ms kbn-bb4-link.telia.net (62.115.139.175)  18.937 ms
 7  nyk-bb2-link.telia.net (80.91.247.121)  122.086 ms nyk-bb2-link.telia.net (80.91.249.29)  108.387 ms nyk-bb1-link.telia.net (62.115.141.97)  150.047 ms
 8  ash-bb4-link.telia.net (80.91.245.98)  116.031 ms ash-bb3-link.telia.net (62.115.138.23)  108.014 ms  107.482 ms
 9  ash-b1-link.telia.net (62.115.115.148)  107.710 ms ash-b1-link.telia.net (62.115.113.207)  111.558 ms ash-b1-link.telia.net (80.91.252.44)  107.792 ms
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
--2015-12-04 22:08:00--  https://dumps.wikimedia.org/other/static_html_dumps/2008-06/it/wikipedia-it-html.tar.7z
Resolving dumps.wikimedia.org... 208.80.154.11, 2620:0:861:1:208:80:154:11
Connecting to dumps.wikimedia.org|208.80.154.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1743914079 (1.6G) [application/x-7z-compressed]
Saving to: “/dev/null”

16% [===============>                                                                                      ] 285,720,335  434K/s  eta 66m 51s

The faster server is on another datacentre (AFAIK) and goes through a slightly different Telia route AFAICT:

$ traceroute dumps.wikimedia.org
traceroute to dumps.wikimedia.org (208.80.154.11), 30 hops max, 60 byte packets
 1  172.31.1.117 (172.31.1.117)  15.915 ms  16.205 ms  16.478 ms
 2  pertr1-te0-0-0-1-1308.pit.hel.fi.nblnet.com (83.145.255.33)  1.236 ms  1.781 ms  2.377 ms
 3  rtr1-po2.pas.hel.fi.nblnet.com (83.145.255.201)  0.500 ms  1.490 ms  0.954 ms
 4  hls-b3-link.telia.net (213.248.84.237)  18.260 ms  18.656 ms  18.885 ms
 5  s-bb3-link.telia.net (62.115.114.46)  8.345 ms s-bb3-link.telia.net (62.115.143.254)  8.572 ms s-bb3-link.telia.net (62.115.113.104)  8.132 ms
 6  kbn-bb4-link.telia.net (62.115.139.62)  14.884 ms kbn-bb3-link.telia.net (62.115.139.60)  17.459 ms kbn-bb3-link.telia.net (62.115.139.167)  17.416 ms
 7  nyk-bb1-link.telia.net (213.155.134.50)  128.121 ms nyk-bb2-link.telia.net (62.115.141.105)  146.857 ms  146.852 ms
 8  ash-bb4-link.telia.net (213.155.134.144)  109.426 ms ash-bb3-link.telia.net (213.155.130.77)  104.156 ms ash-bb4-link.telia.net (62.115.138.29)  112.078 ms
 9  ash-b1-link.telia.net (62.115.113.209)  106.115 ms ash-b1-link.telia.net (62.115.115.148)  110.387 ms ash-b1-link.telia.net (62.115.113.213)  108.959 ms
10  wikimedia-ic-308845-ash-b2.c.telia.net (80.239.132.226)  108.120 ms  110.596 ms  112.190 ms
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Related Objects
Search...

Status	Assigned	Task
Duplicate	ArielGlenn	T122917 Provide a good download service of dumps from Wikimedia
Declined	ArielGlenn	T45647 Sometimes (at peak usage?), dumps.wikimedia.org becomes very slow for users (sometimes unresponsive)
Declined	ArielGlenn	T120425 dumps.wikimedia.org seems to have poor throughput towards some destinations

Event Timeline

Nemo_bis created this task.Dec 4 2015, 9:23 PM

Nemo_bis assigned this task to ArielGlenn.

Nemo_bis raised the priority of this task from to High.

Nemo_bis updated the task description. (Show Details)

Nemo_bis added projects: SRE, Datasets-General-or-Unknown.

Nemo_bis updated the task description. (Show Details)

Nemo_bis set Security to None.

Nemo_bis added subscribers: BBlack, Vituzzu, Aklapper and 19 others.

Dzahn added a project: netops.Dec 4 2015, 9:27 PM

Nemo_bis updated the task description. (Show Details)Dec 4 2015, 9:28 PM

Nemo_bis removed a project: netops.

Nemo_bis added a project: netops.

Can you additionally try:
wget -O /dev/null --no-check-certificate --header='Host: upload.wikimedia.org' https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/f/f2/Wikiconference_in_Moscow_day_2_part_5.webm

Peachey88 unsubscribed.Dec 4 2015, 10:24 PM

Thanks, I tried quickly crafting such a wget line but failed; now trying that one (but it seems that's only slightly faster, around 400-900 KB/s).

Update:

WARNING: certificate common name “*.wikipedia.org” doesn’t match requested host name “upload-lb.eqiad.wikimedia.org”.
HTTP request sent, awaiting response... 200 OK
Length: 3573382210 (3.3G) [video/webm]
Saving to: “/dev/null”

100%[===================================================================================================>] 3,573,382,210 1.25M/s   in 57m 58s

2015-12-07 13:18:49 (1003 KB/s) - “/dev/null” saved [3573382210/3573382210]

In the meanwhile, my personal solution has been to set up a caching web proxy with polipo. It took about 2 h from the moment I started searching information on how to set up a proxy (I had zero knowledge on the topic) to the moment I made my first successful download.

With a 512 MB RAM, 20 GB HD DigitalOcean instance in Amsterdam (0.007 $/h) I'm currently using this configuration for testing (after sudo apt-get install polipo):

/etc/polipo/config:

# This file only needs to list configuration variables that deviate
# from the default values.  See /usr/share/doc/polipo/examples/config.sample
# and "polipo -v" for variables you can tweak and further information.

logSyslog = true
logFile = /var/log/polipo/polipo.log
maxDiskCacheEntrySize = 10737418240
#maxDiskEntries = 32
proxyAddress = ::
allowedPorts = 80
disableLocalInterface = true
objectHighMark = 30
publicObjectLowMark = 16
# dumps.wikimedia.org allows up to 3 connections
serverExpireTime = 14d
serverMaxSlots = 3
serverSlots = 3
serverSlots1 = 3

/etc/polipo/forbidden:

^[^h][^t][^t][^p][^:][^/][^/][^d][^u][^m][^p][^s][^.][^w][^i][^k][^i][^m][^e][^d][^i][^a][^.][^o][^r][^g][^/][^.][^+]

The first download from the slow machine went from some 400 KB/s to some 1.12M/s, and stays at about 1.9 MB/s (dumps.wikimedia.org max bandwidth) on the fast server. The second download finds the file cached and is spectacularly fast on the fast machine (in Helsinki), while still going through Telia:

$ wget http://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-pages-articles.xml.bz2 -e use_proxy=on -e http_proxy=$PUBLIC_IP:8123 -O /dev/null
--2015-12-07 12:20:00--  http://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-pages-articles.xml.bz2
Connecting to $PUBLIC_IP:8123... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2110045606 (2,0G) [application/octet-stream]
Saving to: `/dev/null'

100%[===================================================================================================>] 2 110 045 606 82,8M/s   in 24s

2015-12-07 12:20:24 (82,7 MB/s) - `/dev/null' saved [2110045606/2110045606]

The second download on the slow machine was about as fast as the maximum dumps.wikimedia.org ever gives and 5 times my average without proxy:

Proxy request sent, awaiting response... 200 OK
Length: 2110045606 (2.0G) [application/octet-stream]
Saving to: “/dev/null”

100%[===================================================================================================>] 2,110,045,606 1.87M/s   in 19m 18s

2015-12-07 12:38:17 (1.74 MB/s) - “/dev/null” saved [2110045606/2110045606]

I start believing that making dumps.wikimedia.org faster is hopeless. Just set up a cache in Amsterdam, able to hold at least 1 and at most 10? big files (10-25 GB), with a minimally smart cache eviction system (to hold recently created and popular dumps for a day or two at least), and be done with it. Such a service can probably be handled even by a private with some 10 $/month.

for comparison:
wget http://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-pages-articles.xml.bz2 for me fluctuates between 1 and 1.4MB/sec. my traceroute shows I go through gtt:

[ariel@bigtrouble ~]$ traceroute dumps.wikimedia.org
traceroute to dumps.wikimedia.org (208.80.154.11), 30 hops max, 60 byte packets
1 gateway (192.168.1.1) 3.521 ms 3.513 ms 3.513 ms
2 bbras-llu-kln-15L500.forthnet.gr (213.16.246.12) 27.922 ms 30.084 ms 31.618 ms
3 213.16.247.1 (213.16.247.1) 33.173 ms 33.204 ms 35.537 ms
4 core-kln-12Be2.forthnet.gr (213.16.247.13) 39.802 ms 39.742 ms 39.789 ms
5 212.162.46.109 (212.162.46.109) 52.006 ms 53.547 ms 53.556 ms
6 * * *
7 * * *
8 GTT-level3-4x10G.Frankfurt.Level3.net (4.68.63.58) 83.564 ms 85.118 ms 85.116 ms
9 et-4-1-0.was10.ip4.gtt.net (89.149.182.18) 169.353 ms 177.903 ms 177.925 ms
10 xe-5-3-1.cr2-eqiad.wikimedia.org (173.241.131.218) 183.374 ms 169.323 ms 170.851 ms
11 * * *
12 * * *
13 * * *
etc.

Here's mine (yep, from windows):

1     1 ms     1 ms     1 ms  192.168.0.1
2     2 ms     1 ms     1 ms  192.168.1.1
3     *       47 ms    48 ms  192.168.100.1 [DSLAM]
4    49 ms    49 ms    46 ms  172.18.1.17
5    51 ms    48 ms    53 ms  172.18.0.25
6    66 ms    66 ms    67 ms  172.19.240.181
7     *       94 ms    67 ms  93.186.128.217
8    71 ms    68 ms    65 ms  et-7-1-0-51.milano51.mil.seabone.net [195.22.205.241]
9    83 ms    69 ms    66 ms  ntt-verio.milano51.mil.seabone.net [93.186.128.157]

10 193 ms 191 ms 193 ms ae-20.r03.amstnl02.nl.bb.gin.ntt.net [129.250.2.50]
11 106 ms 123 ms 95 ms ae-4.r23.amstnl02.nl.bb.gin.ntt.net [129.250.2.146]
12 193 ms 202 ms 194 ms ae-5.r23.asbnva02.us.bb.gin.ntt.net [129.250.6.162]
13 191 ms 222 ms 189 ms ae-45.r06.asbnva02.us.bb.gin.ntt.net [129.250.6.11]
14 186 ms 202 ms 173 ms xe-0-6-0-20.r06.asbnva02.us.ce.gin.ntt.net [129.250.204.190]
15 173 ms 173 ms 173 ms dataset1001.wikimedia.org [208.80.154.11]

Two days ago avg. download speed was roughly 200kB, ranging from 400 to 100. It seems to vary according to daily load in carrier btw. I'm on DSL on ATM on PDH but months before I had 350kB/s avg download speed.

Traceroutes *to* dumps provide some insight but are generally not very useful, since you're downloading files and not uploading.

If you can, please paste your IP address (masked to a /24 for IPv4 or /48 for an IPv6) or contact me privately with them, so that we can run reverse traceroutes *from* Wikimedia to you.

(also please keep the polipo and other architecture recommendations out of this bug report)

Sorry i did not mean to make a recommendation, just to show a workaround that anyone can use with few cents. So i can remember what to do next time i encounter the bug on this machine.

Comparison eqiad/esams from nike.fixme.fi to upload.wikimedia.org

$ for c in eqiad esams; do for p in 4 6; do wget -S -O /dev/null -$p --no-check-certificate --header='Host: upload.wikimedia.org' https://upload-lb.$c.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu ; done ; done
--2015-12-07 18:28:37--  https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.eqiad.wikimedia.org... 208.80.154.240
Connecting to upload-lb.eqiad.wikimedia.org|208.80.154.240|:443... connected.
WARNING: certificate common name “*.wikipedia.org” doesn’t match requested host name “upload-lb.eqiad.wikimedia.org”.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx/1.9.4
  Date: Mon, 07 Dec 2015 16:28:48 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: tx1e1f0c4c3661471583dae-005665b3ba
  X-Varnish: 3054394245 3054387028, 2781933600
  Via: 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 6
  X-Cache: cp1051 hit (1), cp1061 frontend miss (0)
  Strict-Transport-Security: max-age=31536000
  Set-Cookie: WMF-Last-Access=07-Dec-2015;Path=/;HttpOnly;Expires=Fri, 08 Jan 2016 12:00:00 GMT
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 109.204.237.158
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: “/dev/null”

100%[=====================================================================================================>] 380,731,035 1.93M/s   in 4m 44s

2015-12-07 18:33:22 (1.28 MB/s) - “/dev/null” saved [380731035/380731035]

--2015-12-07 18:33:22--  https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.eqiad.wikimedia.org... 2620:0:861:ed1a::2:b
Connecting to upload-lb.eqiad.wikimedia.org|2620:0:861:ed1a::2:b|:443... failed: Network is unreachable.
--2015-12-07 18:33:22--  https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.esams.wikimedia.org... 91.198.174.208
Connecting to upload-lb.esams.wikimedia.org|91.198.174.208|:443... connected.
WARNING: certificate common name “*.wikipedia.org” doesn’t match requested host name “upload-lb.esams.wikimedia.org”.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.9.4
  Date: Mon, 07 Dec 2015 16:33:33 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: tx1e1f0c4c3661471583dae-005665b3ba
  X-Varnish: 3054538079 3054387028, 445315921 445090456, 3629406201
  Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 291
  X-Cache: cp1051 hit (3), cp3045 hit (3), cp3045 frontend miss (0)
  Strict-Transport-Security: max-age=31536000
  Set-Cookie: WMF-Last-Access=07-Dec-2015;Path=/;HttpOnly;Expires=Fri, 08 Jan 2016 12:00:00 GMT
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 109.204.237.158
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: “/dev/null”

100%[=====================================================================================================>] 380,731,035 3.05M/s   in 1m 42s  p

2015-12-07 18:35:04 (3.56 MB/s) - “/dev/null” saved [380731035/380731035]

--2015-12-07 18:35:04--  https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.esams.wikimedia.org... 2620:0:862:ed1a::2:b
Connecting to upload-lb.esams.wikimedia.org|2620:0:862:ed1a::2:b|:443... failed: Network is unreachable.

Comparison IPv4/IPv6 and eqiad/esams from koti.kapsi.fi to upload.wikimedia.org

$ wget -S -O /dev/null -4 --no-check-certificate --header='Host: upload.wikimedia.org' https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
--2015-12-07 18:33:31--  https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)... 91.198.174.208
Connecting to upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)|91.198.174.208|:443... connected.
The certificate's owner does not match hostname `upload-lb.esams.wikimedia.org'
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.9.4
  Date: Mon, 07 Dec 2015 16:33:32 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: tx1e1f0c4c3661471583dae-005665b3ba
  X-Varnish: 3054538079 3054387028, 445314335 445090456, 2888369252
  Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 289
  X-Cache: cp1051 hit (3), cp3045 hit (2), cp3034 frontend miss (0)
  Strict-Transport-Security: max-age=31536000
  Set-Cookie: WMF-Last-Access=07-Dec-2015;Path=/;HttpOnly;Expires=Fri, 08 Jan 2016 12:00:00 GMT
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 217.30.184.161
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: `/dev/null'

100%[=====================================================================================================>] 380 731 035 12,9M/s   in 29s

2015-12-07 18:34:01 (12,7 MB/s) - `/dev/null' saved [380731035/380731035]

$ wget -S -O /dev/null -6 --no-check-certificate --header='Host: upload.wikimedia.org' https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
--2015-12-07 18:33:36--  https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)... 2620:0:862:ed1a::2:b
Connecting to upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)|2620:0:862:ed1a::2:b|:443... connected.
The certificate's owner does not match hostname `upload-lb.esams.wikimedia.org'
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.9.4
  Date: Mon, 07 Dec 2015 16:33:37 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: tx1e1f0c4c3661471583dae-005665b3ba
  X-Varnish: 3054538079 3054387028, 445321127 445090456, 2711812112
  Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 294
  X-Cache: cp1051 hit (3), cp3045 hit (4), cp3039 frontend miss (0)
  Strict-Transport-Security: max-age=31536000
  Set-Cookie: WMF-Last-Access=07-Dec-2015;Path=/;HttpOnly;Expires=Fri, 08 Jan 2016 12:00:00 GMT
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 2001:1bc8:1004::1
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: `/dev/null'

100%[=====================================================================================================>] 380 731 035 14,4M/s   in 27s

2015-12-07 18:34:04 (13,5 MB/s) - `/dev/null' saved [380731035/380731035]

$ wget -S -O /dev/null -4 --no-check-certificate --header='Host: upload.wikimedia.org' https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
--2015-12-07 18:33:49--  https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)... 208.80.154.240
Connecting to upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)|208.80.154.240|:443... connected.
The certificate's owner does not match hostname `upload-lb.eqiad.wikimedia.org'
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.9.4
  Date: Mon, 07 Dec 2015 16:33:50 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: tx1e1f0c4c3661471583dae-005665b3ba
  X-Varnish: 3054740501 3054387028, 3019071710
  Via: 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 308
  X-Cache: cp1051 hit (4), cp1050 frontend miss (0)
  Strict-Transport-Security: max-age=31536000
  Set-Cookie: WMF-Last-Access=07-Dec-2015;Path=/;HttpOnly;Expires=Fri, 08 Jan 2016 12:00:00 GMT
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 217.30.184.161
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: `/dev/null'

100%[=====================================================================================================>] 380 731 035 5,97M/s   in 3m 7s

2015-12-07 18:36:57 (1,94 MB/s) - `/dev/null' saved [380731035/380731035]

$ wget -S -O /dev/null -6 --no-check-certificate --header='Host: upload.wikimedia.org' https://upload-lb.eqiad.wikimedia.org/wi
kipedia/commons/9/98/J%C3%A6rnet.djvu
--2015-12-07 18:34:00--  https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)... 2620:0:861:ed1a::2:b
Connecting to upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)|2620:0:861:ed1a::2:b|:443... connected.
The certificate's owner does not match hostname `upload-lb.eqiad.wikimedia.org'
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.9.4
  Date: Mon, 07 Dec 2015 16:34:01 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: tx1e1f0c4c3661471583dae-005665b3ba
  X-Varnish: 3054753263 3054387028, 525201951
  Via: 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 319
  X-Cache: cp1051 hit (6), cp1062 frontend miss (0)
  Strict-Transport-Security: max-age=31536000
  Set-Cookie: WMF-Last-Access=07-Dec-2015;Path=/;HttpOnly;Expires=Fri, 08 Jan 2016 12:00:00 GMT
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 2001:1bc8:1004::1
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: `/dev/null'

100%[=====================================================================================================>] 380 731 035 1,08M/s   in 2m 2s

2015-12-07 18:36:03 (2,98 MB/s) - `/dev/null' saved [380731035/380731035]

Comparison IPv4 vs. IPv6, koti.kapsi.fi vs. nike.fixme.fi, big vs. small file, to dumps.wikimedia.org

$ wget -O /dev/null -S -4 https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-externallinks.sql.gz
--2015-12-07 18:13:04--  https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-externallinks.sql.gz
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 208.80.154.11
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|208.80.154.11|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.1.19
  Date: Mon, 07 Dec 2015 16:13:05 GMT
  Content-Type: application/octet-stream
  Content-Length: 200447836
  Last-Modified: Thu, 03 Dec 2015 19:00:03 GMT
  Connection: keep-alive
  Accept-Ranges: bytes
Length: 200447836 (191M) [application/octet-stream]
Saving to: `/dev/null'

100%[=====================================================================================================>] 200 447 836 1,57M/s   in 1m 48s

2015-12-07 18:14:53 (1,77 MB/s) - `/dev/null' saved [200447836/200447836]
$ wget -O /dev/null -S -4 https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-pages-articles.xml.bz2
--2015-12-07 18:15:19--  https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-pages-articles.xml.bz2
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 208.80.154.11
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|208.80.154.11|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.1.19
  Date: Mon, 07 Dec 2015 16:15:20 GMT
  Content-Type: application/octet-stream
  Content-Length: 2110045606
  Last-Modified: Fri, 04 Dec 2015 00:19:28 GMT
  Connection: keep-alive
  Accept-Ranges: bytes
Length: 2110045606 (2,0G) [application/octet-stream]
Saving to: `/dev/null'

100%[===================================================================================================>] 2 110 045 606  655K/s   in 26m 14s

2015-12-07 18:41:34 (1,28 MB/s) - `/dev/null' saved [2110045606/2110045606]

$ wget -O /dev/null -S -6 https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-externallinks.sql.gz
--2015-12-07 18:13:36--  https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-externallinks.sql.gz
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620:0:861:1:208:80:154:11
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620:0:861:1:208:80:154:11|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.1.19
  Date: Mon, 07 Dec 2015 16:13:37 GMT
  Content-Type: application/octet-stream
  Content-Length: 200447836
  Last-Modified: Thu, 03 Dec 2015 19:00:03 GMT
  Connection: keep-alive
  Accept-Ranges: bytes
Length: 200447836 (191M) [application/octet-stream]
Saving to: `/dev/null'

100%[=====================================================================================================>] 200 447 836 1,40M/s   in 2m 6s

2015-12-07 18:15:43 (1,52 MB/s) - `/dev/null' saved [200447836/200447836]

$ wget -O /dev/null -S -6 https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-pages-articles.xml.bz2
--2015-12-07 18:16:11--  https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-pages-articles.xml.bz2
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620:0:861:1:208:80:154:11
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620:0:861:1:208:80:154:11|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.1.19
  Date: Mon, 07 Dec 2015 16:16:12 GMT
  Content-Type: application/octet-stream
  Content-Length: 2110045606
  Last-Modified: Fri, 04 Dec 2015 00:19:28 GMT
  Connection: keep-alive
  Accept-Ranges: bytes
Length: 2110045606 (2,0G) [application/octet-stream]
Saving to: `/dev/null'

100%[===================================================================================================>] 2 110 045 606 1,87M/s   in 19m 29s

2015-12-07 18:35:41 (1,72 MB/s) - `/dev/null' saved [2110045606/2110045606]

nike.fixme.fi has no IPv6 and is too slow to try the big file:

$ wget -O /dev/null -S -6 https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-externallinks.sql.gz
--2015-12-07 18:16:21--  https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-externallinks.sql.gz
Resolving dumps.wikimedia.org... 2620:0:861:1:208:80:154:11
Connecting to dumps.wikimedia.org|2620:0:861:1:208:80:154:11|:443... failed: Network is unreachable.

$ wget -O /dev/null -S -4 https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-externallinks.sql.gz
--2015-12-07 18:16:28--  https://dumps.wikimedia.org/itwiki/20151202/itwiki-20151202-externallinks.sql.gz
Resolving dumps.wikimedia.org... 208.80.154.11
Connecting to dumps.wikimedia.org|208.80.154.11|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx/1.1.19
  Date: Mon, 07 Dec 2015 16:16:40 GMT
  Content-Type: application/octet-stream
  Content-Length: 200447836
  Last-Modified: Thu, 03 Dec 2015 19:00:03 GMT
  Connection: keep-alive
  Accept-Ranges: bytes
Length: 200447836 (191M) [application/octet-stream]
Saving to: “/dev/null”

100%[=====================================================================================================>] 200,447,836  366K/s   in 8m 30s  

2015-12-07 18:24:59 (384 KB/s) - “/dev/null” saved [200447836/200447836]

An IP from Milan for comparison: 1 vs. 6 MB/s

No IPv6 connection but a good bandwidth (at least 12 MiB/s).

$ for c in eqiad esams; do for p in 4 6; do wget -S -O /dev/null -$p --no-check-certificate --header='Host: upload.wikimedia.org' https://upload-lb.$c.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu ; done ; done
--2015-12-13 22:11:15--  https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Risoluzione di upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)... 208.80.154.240
Connessione a upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)|208.80.154.240|:443... connesso.
AVVERTIMENTO: nessuno dei nomi alternativi indicati nel certificato corresponde al
    nome dell'host richiesto "upload-lb.eqiad.wikimedia.org".
Richiesta HTTP inviata, in attesa di risposta... 
  HTTP/1.1 200 OK
  Server: nginx/1.9.4
  Date: Sun, 13 Dec 2015 21:11:15 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: txbcca82e36af9491bbfa81-00566ddef3
  X-Varnish: 3670640299, 3875762258
  Via: 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 0
  X-Cache: cp1051 miss (0), cp1074 frontend miss (0)
  Strict-Transport-Security: max-age=31536000
  Set-Cookie: WMF-Last-Access=13-Dec-2015;Path=/;HttpOnly;Expires=Thu, 14 Jan 2016 12:00:00 GMT
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 80.169.235.78
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Lunghezza: 380731035 (363M) [image/vnd.djvu]                                                                                                                                     
Salvataggio in: "/dev/null"                                                                                                                                                      
                                                                                                                                                                                 
/dev/null                                    100%[=============================================================================================>] 363,09M  1,70MB/s   in 7m 55s  
                                                                                                                                                                                 
2015-12-13 22:19:11 (782 KB/s) - "/dev/null" salvato [380731035/380731035]                                                                                                       
                                                                                                                                                                                 
--2015-12-13 22:19:11--  https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu                                                                           
Risoluzione di upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)... 2620:0:861:ed1a::2:b                                                                             
Connessione a upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)|2620:0:861:ed1a::2:b|:443... non riuscito: La rete non è raggiungibile.                              
--2015-12-13 22:19:11--  https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu                                                                           
Risoluzione di upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)... 91.198.174.208                                                                                   
Connessione a upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)|91.198.174.208|:443... connesso.
AVVERTIMENTO: nessuno dei nomi alternativi indicati nel certificato corresponde al
    nome dell'host richiesto "upload-lb.esams.wikimedia.org".
Richiesta HTTP inviata, in attesa di risposta... 
  HTTP/1.1 200 OK
  Server: nginx/1.9.4
  Date: Sun, 13 Dec 2015 21:19:11 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: txbcca82e36af9491bbfa81-00566ddef3
  X-Varnish: 3671205973 3670640299, 1106458678, 351256311
  Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 476
  X-Cache: cp1051 hit (1), cp3045 miss (0), cp3045 frontend miss (0)
  Strict-Transport-Security: max-age=31536000
  Set-Cookie: WMF-Last-Access=13-Dec-2015;Path=/;HttpOnly;Expires=Thu, 14 Jan 2016 12:00:00 GMT
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 80.169.235.78
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Lunghezza: 380731035 (363M) [image/vnd.djvu]
Salvataggio in: "/dev/null"

/dev/null                                    100%[=============================================================================================>] 363,09M  2,64MB/s   in 65s    

2015-12-13 22:20:16 (5,56 MB/s) - "/dev/null" salvato [380731035/380731035]

--2015-12-13 22:20:16--  https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Risoluzione di upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)... 2620:0:862:ed1a::2:b
Connessione a upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)|2620:0:862:ed1a::2:b|:443... non riuscito: La rete non è raggiungibile.

ArielGlenn added a parent task: T122917: Provide a good download service of dumps from Wikimedia.Jan 5 2016, 8:16 PM

ArielGlenn moved this task from Backlog to Up Next on the Datasets-General-or-Unknown board.Feb 3 2016, 3:42 PM

Still an issue?

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptJul 8 2016, 11:11 PM

Yes AFAICT, from a quick check. What makes you think it wouldn't be? Ops, if some specific testing is (still) needed please specify it.

ArielGlenn moved this task from Up Next to Backlog on the Datasets-General-or-Unknown board.Jul 19 2016, 1:13 PM

mark renamed this task from dumps.wikimedia.org seems to have poor networking towards Telia to dumps.wikimedia.org seems to have poor throughput towards some destinations.Jul 27 2016, 5:39 PM

mark lowered the priority of this task from High to Medium.

mark subscribed.Jul 27 2016, 5:47 PM

Thanks mark for clarifying the subject.

In T120425#2444215, @Nemo_bis wrote:

Yes AFAICT, from a quick check. What makes you think it wouldn't be? Ops, if some specific testing is (still) needed please specify it.

@Nemo_bis: see Faidon's reply on Dec 7; if you provide us with your IP addresses we can take a look at the routing in the (more important) other direction to see if we can correlate some things. Unless we get that, this ticket is not very actionable I'm afraid.

In T120425#2501557, @mark wrote:

@Nemo_bis: see Faidon's reply on Dec 7; if you provide us with your IP addresses we can take a look at the routing in the (more important) other direction to see if we can correlate some things. Unless we get that, this ticket is not very actionable I'm afraid.

I provided my IP when I was asked. I can provide it multiple times, of course, although every single host I use has the same issues.

This is a test of download from upload.wm.o on 217.30.184.184 (lakka.kapsi.fi). Just dumping here

federico@lakka:~$ for c in eqiad esams; do for p in 4 6; do wget -S -O /dev/null -$p --no-check-certificate --header='Host: upload.wikimedia.org' https://upload-lb.$c.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu ; done ; done
--2016-08-02 14:45:06--  https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)... 208.80.154.240
Connecting to upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)|208.80.154.240|:443... connected.
The certificate's owner does not match hostname ‘upload-lb.eqiad.wikimedia.org’
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 02 Aug 2016 11:45:06 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: txd45aff4ab3e840d0bbbbd-0057a08784
  X-Varnish: 2230262755 2230114928, 2356874202
  Via: 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 62
  X-Cache: cp1074 hit/1, cp1063 pass
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 217.30.184.161
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: ‘/dev/null’

/dev/null                             0%[                                                                     ]   3,50M  3,93MB/s              /dev/null                             2%[>                                                                    ]   7,49M  6,67MB/s              /dev/null                             2%[=>                                                                   ]  10,61M  7,78MB/s              /dev/null                             4%[=>                                                                   ]  15,16M  9,56MB/s              /dev/null                             5%[==>                                                                  ]  19,64M  10,9MB/s              /dev/null                             6%[===>                                                                 ]  24,18M  11,9MB/s              /dev/null                             7%[====>                                                                ]  28,79M  12,8MB/s              /dev/null                             9%[=====>                                                               ]  33,45M  13,6MB/s              /dev/null                            10%[======>                                                              ]  38,17M  14,2MB/s              /dev/null                            11%[=======>                                                             ]  42,95M  14,8MB/s              /dev/null                            13%[========>                                                            ]  47,78M  15,3MB/s   eta 21s    /dev/null                            14%[=========>                                                           ]  52,66M  15,7MB/s   eta 21s    /dev/null                            15%[=========>                                                           ]  57,58M  16,1MB/s   eta 21s    /dev/null                            17%[==========>                                                          ]  62,55M  17,2MB/s   eta 21s    /dev/null                            18%[===========>                                                         ]  67,54M  18,3MB/s   eta 21s    /dev/null                            19%[============>                                                        ]  72,21M  19,8MB/s   eta 17s    /dev/null                            20%[=============>                                                       ]  75,73M  19,7MB/s   eta 17s    /dev/null                            21%[==============>                                                      ]  79,35M  20,2MB/s   eta 17s    /dev/null                            22%[==============>                                                      ]  83,03M  20,0MB/s   eta 17s    /dev/null                            23%[===============>                                                     ]  86,76M  20,0MB/s   eta 17s    /dev/null                            24%[================>                                                    ]  90,54M  19,5MB/s   eta 16s    /dev/null                            25%[================>                                                    ]  94,33M  19,2MB/s   eta 16s    /dev/null                            27%[=================>                                                   ]  98,16M  19,3MB/s   eta 16s    /dev/null                            28%[==================>                                                  ] 102,03M  18,8MB/s   eta 16s    /dev/null                            29%[===================>                                                 ] 105,94M  18,5MB/s   eta 16s    /dev/null                            30%[===================>                                                 ] 109,91M  18,6MB/s   eta 15s    /dev/null                            31%[====================>                                                ] 113,65M  18,0MB/s   eta 15s    /dev/null                            32%[=====================>                                               ] 116,47M  17,8MB/s   eta 15s    /dev/null                            32%[=====================>                                               ] 119,33M  17,1MB/s   eta 15s    /dev/null                            33%[======================>                                              ] 122,24M  16,2MB/s   eta 15s    /dev/null                            34%[======================>                                              ] 125,17M  15,9MB/s   eta 14s    /dev/null                            35%[=======================>                                             ] 128,16M  15,4MB/s   eta 14s    /dev/null                            36%[=======================>                                             ] 131,17M  15,6MB/s   eta 14s    /dev/null                            36%[========================>                                            ] 134,21M  15,3MB/s   eta 14s    /dev/null                            37%[=========================>                                           ] 137,30M  14,9MB/s   eta 14s    /dev/null                            38%[=========================>                                           ] 140,42M  15,0MB/s   eta 14s    /dev/null                            39%[==========================>                                          ] 143,57M  14,8MB/s   eta 14s    /dev/null                            40%[==========================>                                          ] 146,75M  14,3MB/s   eta 14s    /dev/null                            41%[===========================>                                         ] 149,95M  14,5MB/s   eta 14s    /dev/null                            42%[============================>                                        ] 153,16M  14,2MB/s   eta 14s    /dev/null                            43%[============================>                                        ] 156,39M  13,7MB/s   eta 13s    /dev/null                            43%[=============================>                                       ] 159,65M  14,1MB/s   eta 13s    /dev/null                            44%[=============================>                                       ] 162,93M  13,9MB/s   eta 13s    /dev/null                            45%[==============================>                                      ] 166,24M  13,8MB/s   eta 13s    /dev/null                            46%[===============================>                                     ] 169,54M  14,1MB/s   eta 13s    /dev/null                            47%[===============================>                                     ] 172,86M  14,3MB/s   eta 12s    /dev/null                            48%[================================>                                    ] 176,19M  14,2MB/s   eta 12s    /dev/null                            49%[=================================>                                   ] 179,53M  14,5MB/s   eta 12s    /dev/null                            50%[=================================>                                   ] 182,87M  14,6MB/s   eta 12s    /dev/null                            51%[==================================>                                  ] 186,23M  14,4MB/s   eta 12s    /dev/null                            52%[===================================>                                 ] 189,60M  14,7MB/s   eta 11s    /dev/null                            53%[===================================>                                 ] 192,96M  14,8MB/s   eta 11s    /dev/null                            54%[====================================>                                ] 196,33M  14,6MB/s   eta 11s    /dev/null                            55%[====================================>                                ] 199,71M  14,9MB/s   eta 11s    /dev/null                            55%[=====================================>                               ] 203,08M  15,0MB/s   eta 11s    /dev/null                            56%[======================================>                              ] 206,48M  14,8MB/s   eta 10s    /dev/null                            57%[======================================>                              ] 209,86M  15,0MB/s   eta 10s    /dev/null                            58%[=======================================>                             ] 213,25M  15,1MB/s   eta 10s    /dev/null                            59%[========================================>                            ] 216,64M  14,9MB/s   eta 10s    /dev/null                            60%[========================================>                            ] 220,03M  15,1MB/s   eta 10s    /dev/null                            61%[=========================================>                           ] 223,42M  15,2MB/s   eta 9s     /dev/null                            62%[==========================================>                          ] 226,80M  14,9MB/s   eta 9s     /dev/null                            63%[==========================================>                          ] 230,19M  15,2MB/s   eta 9s     /dev/null                            64%[===========================================>                         ] 233,59M  15,2MB/s   eta 9s     /dev/null                            65%[============================================>                        ] 236,97M  15,0MB/s   eta 9s     /dev/null                            66%[============================================>                        ] 240,34M  15,2MB/s   eta 8s     /dev/null                            67%[=============================================>                       ] 243,72M  15,2MB/s   eta 8s     /dev/null                            68%[=============================================>                       ] 247,10M  15,0MB/s   eta 8s     /dev/null                            68%[==============================================>                      ] 250,48M  15,2MB/s   eta 8s     /dev/null                            69%[===============================================>                     ] 253,88M  15,2MB/s   eta 8s     /dev/null                            70%[===============================================>                     ] 257,28M  15,0MB/s   eta 7s     /dev/null                            71%[================================================>                    ] 260,67M  15,0MB/s   eta 7s     /dev/null                            72%[=================================================>                   ] 264,07M  15,2MB/s   eta 7s     /dev/null                            73%[=================================================>                   ] 267,47M  15,1MB/s   eta 7s     /dev/null                            74%[==================================================>                  ] 270,87M  15,1MB/s   eta 7s     /dev/null                            75%[===================================================>                 ] 274,29M  15,2MB/s   eta 6s     /dev/null                            76%[===================================================>                 ] 277,70M  15,3MB/s   eta 6s     /dev/null                            77%[====================================================>                ] 281,12M  15,1MB/s   eta 6s     /dev/null                            78%[=====================================================>               ] 284,54M  15,1MB/s   eta 6s     /dev/null                            79%[=====================================================>               ] 287,97M  15,3MB/s   eta 6s     /dev/null                            80%[======================================================>              ] 291,43M  15,1MB/s   eta 5s     /dev/null                            81%[=======================================================>             ] 294,89M  15,1MB/s   eta 5s     /dev/null                            82%[=======================================================>             ] 298,35M  15,2MB/s   eta 5s     /dev/null                            83%[========================================================>            ] 301,85M  15,2MB/s   eta 5s     /dev/null                            84%[=========================================================>           ] 305,33M  15,4MB/s   eta 5s     /dev/null                            85%[=========================================================>           ] 308,85M  15,5MB/s   eta 3s     /dev/null                            86%[==========================================================>          ] 312,37M  15,2MB/s   eta 3s     /dev/null                            87%[===========================================================>         ] 315,91M  15,4MB/s   eta 3s     /dev/null                            87%[===========================================================>         ] 319,48M  15,4MB/s   eta 3s     /dev/null                            88%[============================================================>        ] 323,07M  15,7MB/s   eta 3s     /dev/null                            89%[=============================================================>       ] 326,68M  15,6MB/s   eta 2s     /dev/null                            90%[=============================================================>       ] 330,32M  15,8MB/s   eta 2s     /dev/null                            91%[==============================================================>      ] 333,99M  15,9MB/s   eta 2s     /dev/null                            93%[===============================================================>     ] 337,69M  15,8MB/s   eta 2s     /dev/null                            94%[===============================================================>     ] 341,44M  15,9MB/s   eta 2s     /dev/null                            95%[================================================================>    ] 345,21M  16,0MB/s   eta 1s     /dev/null                            96%[=================================================================>   ] 349,02M  16,2MB/s   eta 1s     /dev/null                            97%[==================================================================>  ] 352,87M  16,3MB/s   eta 1s     /dev/null                            98%[==================================================================>  ] 356,76M  16,5MB/s   eta 1s     /dev/null                            99%[===================================================================> ] 360,69M  16,7MB/s   eta 1s     /dev/null                           100%[====================================================================>] 363,09M  16,7MB/s   in 23s      

2016-08-02 14:45:30 (15,7 MB/s) - ‘/dev/null’ saved [380731035/380731035]

--2016-08-02 14:45:30--  https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)... 2620:0:861:ed1a::2:b
Connecting to upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)|2620:0:861:ed1a::2:b|:443... connected.
The certificate's owner does not match hostname ‘upload-lb.eqiad.wikimedia.org’
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 02 Aug 2016 11:45:30 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: txd45aff4ab3e840d0bbbbd-0057a08784
  X-Varnish: 2230329894 2230114928, 3825830785
  Via: 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 86
  X-Cache: cp1074 hit/2, cp1071 pass
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 2001:1bc8:1004::1
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: ‘/dev/null’

/dev/null                             0%[                                                                     ]   3,41M  3,99MB/s              /dev/null                             2%[>                                                                    ]   9,14M  8,58MB/s              /dev/null                             4%[=>                                                                   ]  14,98M  11,7MB/s              /dev/null                             5%[==>                                                                  ]  20,83M  13,9MB/s              /dev/null                             7%[====>                                                                ]  26,73M  15,7MB/s              /dev/null                             8%[=====>                                                               ]  32,58M  17,0MB/s              /dev/null                            10%[======>                                                              ]  37,80M  17,4MB/s              /dev/null                            11%[=======>                                                             ]  42,27M  17,7MB/s              /dev/null                            12%[=======>                                                             ]  45,24M  17,4MB/s              /dev/null                            13%[========>                                                            ]  48,42M  17,1MB/s              /dev/null                            14%[=========>                                                           ]  52,85M  16,7MB/s   eta 19s    /dev/null                            15%[=========>                                                           ]  57,28M  17,0MB/s   eta 19s    /dev/null                            16%[==========>                                                          ]  60,26M  16,8MB/s   eta 19s    /dev/null                            17%[===========>                                                         ]  63,26M  16,7MB/s   eta 19s    /dev/null                            18%[===========>                                                         ]  66,31M  17,5MB/s   eta 19s    /dev/null                            19%[============>                                                        ]  69,37M  18,8MB/s   eta 18s    /dev/null                            19%[============>                                                        ]  72,44M  19,2MB/s   eta 18s    /dev/null                            20%[=============>                                                       ]  75,59M  18,7MB/s   eta 18s    /dev/null                            21%[=============>                                                       ]  78,78M  18,2MB/s   eta 18s    /dev/null                            22%[==============>                                                      ]  81,99M  17,6MB/s   eta 18s    /dev/null                            23%[===============>                                                     ]  85,27M  16,6MB/s   eta 17s    /dev/null                            24%[===============>                                                     ]  88,55M  16,1MB/s   eta 17s    /dev/null                            25%[================>                                                    ]  91,87M  15,5MB/s   eta 17s    /dev/null                            26%[=================>                                                   ]  95,20M  15,1MB/s   eta 17s    /dev/null                            27%[=================>                                                   ]  98,51M  15,0MB/s   eta 17s    /dev/null                            28%[==================>                                                  ] 101,89M  15,0MB/s   eta 16s    /dev/null                            28%[===================>                                                 ] 105,25M  14,8MB/s   eta 16s    /dev/null                            29%[===================>                                                 ] 108,63M  15,0MB/s   eta 16s    /dev/null                            30%[====================>                                                ] 112,06M  14,9MB/s   eta 16s    /dev/null                            31%[====================>                                                ] 115,49M  15,3MB/s   eta 16s    /dev/null                            32%[=====================>                                               ] 118,95M  15,4MB/s   eta 15s    /dev/null                            33%[======================>                                              ] 122,40M  15,3MB/s   eta 15s    /dev/null                            34%[======================>                                              ] 125,83M  15,6MB/s   eta 15s    /dev/null                            35%[=======================>                                             ] 129,29M  15,6MB/s   eta 15s    /dev/null                            36%[========================>                                            ] 132,79M  15,6MB/s   eta 15s    /dev/null                            37%[========================>                                            ] 136,30M  15,9MB/s   eta 14s    /dev/null                            38%[=========================>                                           ] 139,78M  15,9MB/s   eta 14s    /dev/null                            39%[==========================>                                          ] 143,32M  15,9MB/s   eta 14s    /dev/null                            40%[==========================>                                          ] 146,85M  16,1MB/s   eta 14s    /dev/null                            41%[===========================>                                         ] 150,38M  16,0MB/s   eta 14s    /dev/null                            42%[============================>                                        ] 153,87M  16,0MB/s   eta 13s    /dev/null                            43%[============================>                                        ] 157,38M  16,3MB/s   eta 13s    /dev/null                            44%[=============================>                                       ] 160,91M  16,1MB/s   eta 13s    /dev/null                            45%[==============================>                                      ] 164,45M  16,1MB/s   eta 13s    /dev/null                            46%[==============================>                                      ] 167,99M  16,4MB/s   eta 13s    /dev/null                            47%[===============================>                                     ] 171,50M  16,4MB/s   eta 12s    /dev/null                            48%[================================>                                    ] 175,03M  16,2MB/s   eta 12s    /dev/null                            49%[================================>                                    ] 178,55M  16,4MB/s   eta 12s    /dev/null                            50%[=================================>                                   ] 182,08M  16,5MB/s   eta 12s    /dev/null                            51%[==================================>                                  ] 185,59M  16,3MB/s   eta 12s    /dev/null                            52%[==================================>                                  ] 189,12M  16,5MB/s   eta 11s    /dev/null                            53%[===================================>                                 ] 192,66M  16,3MB/s   eta 11s    /dev/null                            54%[====================================>                                ] 196,19M  16,3MB/s   eta 11s    /dev/null                            54%[====================================>                                ] 199,69M  16,5MB/s   eta 11s    /dev/null                            55%[=====================================>                               ] 203,22M  16,3MB/s   eta 11s    /dev/null                            56%[======================================>                              ] 206,75M  16,3MB/s   eta 10s    /dev/null                            57%[======================================>                              ] 210,28M  16,5MB/s   eta 10s    /dev/null                            58%[=======================================>                             ] 213,82M  16,5MB/s   eta 10s    /dev/null                            59%[========================================>                            ] 217,35M  16,3MB/s   eta 10s    /dev/null                            60%[========================================>                            ] 220,83M  16,5MB/s   eta 10s    /dev/null                            61%[=========================================>                           ] 224,37M  16,3MB/s   eta 9s     /dev/null                            62%[==========================================>                          ] 227,92M  16,3MB/s   eta 9s     /dev/null                            63%[==========================================>                          ] 231,48M  16,5MB/s   eta 9s     /dev/null                            64%[===========================================>                         ] 235,03M  16,5MB/s   eta 9s     /dev/null                            65%[============================================>                        ] 238,58M  16,3MB/s   eta 9s     /dev/null                            66%[=============================================>                       ] 242,13M  16,5MB/s   eta 7s     /dev/null                            67%[=============================================>                       ] 245,69M  16,5MB/s   eta 7s     /dev/null                            68%[==============================================>                      ] 249,24M  16,4MB/s   eta 7s     /dev/null                            69%[===============================================>                     ] 252,81M  16,6MB/s   eta 7s     /dev/null                            70%[===============================================>                     ] 256,41M  16,4MB/s   eta 7s     /dev/null                            71%[================================================>                    ] 260,01M  16,4MB/s   eta 6s     /dev/null                            72%[=================================================>                   ] 263,65M  16,7MB/s   eta 6s     /dev/null                            73%[=================================================>                   ] 267,28M  16,5MB/s   eta 6s     /dev/null                            74%[==================================================>                  ] 270,93M  16,4MB/s   eta 6s     /dev/null                            75%[===================================================>                 ] 274,60M  16,7MB/s   eta 6s     /dev/null                            76%[===================================================>                 ] 278,28M  16,8MB/s   eta 5s     /dev/null                            77%[====================================================>                ] 281,97M  16,6MB/s   eta 5s     /dev/null                            78%[=====================================================>               ] 285,69M  16,9MB/s   eta 5s     /dev/null                            79%[======================================================>              ] 289,44M  17,0MB/s   eta 5s     /dev/null                            80%[======================================================>              ] 293,26M  16,8MB/s   eta 5s     /dev/null                            81%[=======================================================>             ] 297,05M  17,1MB/s   eta 4s     /dev/null                            82%[========================================================>            ] 300,89M  17,2MB/s   eta 4s     /dev/null                            83%[========================================================>            ] 304,75M  17,0MB/s   eta 4s     /dev/null                            85%[=========================================================>           ] 308,65M  17,4MB/s   eta 4s     /dev/null                            86%[==========================================================>          ] 312,60M  17,4MB/s   eta 4s     /dev/null                            87%[===========================================================>         ] 316,56M  17,3MB/s   eta 3s     /dev/null                            88%[===========================================================>         ] 320,58M  17,7MB/s   eta 3s     /dev/null                            89%[============================================================>        ] 324,64M  17,9MB/s   eta 3s     /dev/null                            90%[=============================================================>       ] 328,78M  17,8MB/s   eta 3s     /dev/null                            91%[==============================================================>      ] 332,95M  18,2MB/s   eta 3s     /dev/null                            92%[===============================================================>     ] 337,13M  18,4MB/s   eta 2s     /dev/null                            94%[===============================================================>     ] 341,37M  18,2MB/s   eta 2s     /dev/null                            95%[================================================================>    ] 345,66M  18,7MB/s   eta 2s     /dev/null                            96%[=================================================================>   ] 350,02M  18,9MB/s   eta 2s     /dev/null                            97%[==================================================================>  ] 354,42M  18,8MB/s   eta 2s     /dev/null                            98%[===================================================================> ] 358,89M  19,2MB/s   eta 0s     /dev/null                           100%[====================================================================>] 363,09M  19,6MB/s   in 21s      

2016-08-02 14:45:52 (16,9 MB/s) - ‘/dev/null’ saved [380731035/380731035]

--2016-08-02 14:45:52--  https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)... 91.198.174.208
Connecting to upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)|91.198.174.208|:443... connected.
The certificate's owner does not match hostname ‘upload-lb.esams.wikimedia.org’
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 02 Aug 2016 11:45:52 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: txd45aff4ab3e840d0bbbbd-0057a08784
  X-Varnish: 2230380404 2230114928, 3148406992, 2054925723
  Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 108
  X-Cache: cp1074 hit/3, cp3045 miss, cp3046 pass
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 217.30.184.161
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: ‘/dev/null’

/dev/null                             0%[                                                                     ]   1,50M  7,42MB/s              /dev/null                             1%[                                                                     ]   3,92M  9,58MB/s              /dev/null                             1%[>                                                                    ]   6,93M  11,2MB/s              /dev/null                             2%[>                                                                    ]   9,47M  11,5MB/s              /dev/null                             3%[=>                                                                   ]  12,35M  12,1MB/s              /dev/null                             4%[=>                                                                   ]  15,29M  12,3MB/s              /dev/null                             4%[==>                                                                  ]  18,13M  12,6MB/s              /dev/null                             5%[===>                                                                 ]  21,34M  12,9MB/s              /dev/null                             6%[===>                                                                 ]  24,30M  13,1MB/s              /dev/null                             7%[====>                                                                ]  27,60M  13,3MB/s              /dev/null                             8%[====>                                                                ]  30,59M  13,5MB/s              /dev/null                             8%[=====>                                                               ]  31,63M  12,6MB/s              /dev/null                             9%[=====>                                                               ]  34,80M  12,8MB/s              /dev/null                            10%[======>                                                              ]  38,35M  13,1MB/s              /dev/null                            11%[======>                                                              ]  41,50M  13,2MB/s   eta 24s    /dev/null                            12%[=======>                                                             ]  45,01M  13,5MB/s   eta 24s    /dev/null                            13%[========>                                                            ]  48,33M  14,0MB/s   eta 24s    /dev/null                            14%[========>                                                            ]  51,82M  14,3MB/s   eta 24s    /dev/null                            15%[=========>                                                           ]  55,29M  14,5MB/s   eta 24s    /dev/null                            16%[==========>                                                          ]  58,57M  14,7MB/s   eta 22s    /dev/null                            17%[==========>                                                          ]  62,07M  14,9MB/s   eta 22s    /dev/null                            18%[===========>                                                         ]  65,52M  15,0MB/s   eta 22s    /dev/null                            18%[============>                                                        ]  68,94M  15,2MB/s   eta 22s    /dev/null                            19%[============>                                                        ]  72,38M  15,3MB/s   eta 22s    /dev/null                            20%[=============>                                                       ]  75,92M  15,5MB/s   eta 20s    /dev/null                            21%[==============>                                                      ]  79,55M  15,7MB/s   eta 20s    /dev/null                            22%[==============>                                                      ]  83,08M  15,8MB/s   eta 20s    /dev/null                            23%[===============>                                                     ]  86,71M  16,9MB/s   eta 20s    /dev/null                            24%[================>                                                    ]  90,26M  17,1MB/s   eta 20s    /dev/null                            25%[================>                                                    ]  93,76M  17,1MB/s   eta 18s    /dev/null                            26%[=================>                                                   ]  97,32M  17,2MB/s   eta 18s    /dev/null                            27%[==================>                                                  ] 100,92M  17,3MB/s   eta 18s    /dev/null                            28%[==================>                                                  ] 104,61M  17,4MB/s   eta 18s    /dev/null                            29%[===================>                                                 ] 108,22M  17,5MB/s   eta 18s    /dev/null                            30%[====================>                                                ] 111,68M  17,5MB/s   eta 16s    /dev/null                            31%[====================>                                                ] 115,28M  17,7MB/s   eta 16s    /dev/null                            32%[=====================>                                               ] 117,88M  16,0MB/s   eta 16s    /dev/null                            33%[=====================>                                               ] 120,58M  15,6MB/s   eta 16s    /dev/null                            34%[======================>                                              ] 124,38M  15,7MB/s   eta 16s    /dev/null                            35%[=======================>                                             ] 127,51M  15,5MB/s   eta 16s    /dev/null                            36%[========================>                                            ] 131,60M  15,7MB/s   eta 16s    /dev/null                            37%[========================>                                            ] 135,00M  15,6MB/s   eta 16s    /dev/null                            38%[=========================>                                           ] 139,09M  15,8MB/s   eta 16s    /dev/null                            39%[==========================>                                          ] 142,48M  15,6MB/s   eta 14s    /dev/null                            40%[==========================>                                          ] 146,57M  15,8MB/s   eta 14s    /dev/null                            41%[===========================>                                         ] 149,97M  15,7MB/s   eta 14s    /dev/null                            42%[============================>                                        ] 154,06M  15,9MB/s   eta 14s    /dev/null                            43%[============================>                                        ] 157,46M  15,7MB/s   eta 14s    /dev/null                            44%[=============================>                                       ] 160,84M  15,7MB/s   eta 13s    /dev/null                            45%[==============================>                                      ] 164,65M  15,8MB/s   eta 13s    /dev/null                            46%[==============================>                                      ] 168,05M  15,7MB/s   eta 13s    /dev/null                            47%[===============================>                                     ] 171,57M  15,7MB/s   eta 13s    /dev/null                            48%[================================>                                    ] 175,25M  17,3MB/s   eta 13s    /dev/null                            49%[================================>                                    ] 178,88M  17,9MB/s   eta 12s    /dev/null                            50%[=================================>                                   ] 182,47M  18,0MB/s   eta 12s    /dev/null                            51%[==================================>                                  ] 186,32M  18,0MB/s   eta 12s    /dev/null                            52%[===================================>                                 ] 189,78M  17,9MB/s   eta 12s    /dev/null                            53%[===================================>                                 ] 193,59M  18,0MB/s   eta 12s    /dev/null                            54%[====================================>                                ] 197,17M  18,0MB/s   eta 10s    /dev/null                            55%[=====================================>                               ] 200,82M  18,0MB/s   eta 10s    /dev/null                            56%[=====================================>                               ] 204,57M  18,0MB/s   eta 10s    /dev/null                            56%[======================================>                              ] 205,94M  16,7MB/s   eta 10s    /dev/null                            57%[======================================>                              ] 208,08M  16,1MB/s   eta 10s    /dev/null                            58%[=======================================>                             ] 211,12M  16,1MB/s   eta 10s    /dev/null                            59%[=======================================>                             ] 214,80M  15,9MB/s   eta 10s    /dev/null                            60%[========================================>                            ] 218,79M  16,1MB/s   eta 10s    /dev/null                            61%[=========================================>                           ] 222,51M  16,2MB/s   eta 10s    /dev/null                            62%[==========================================>                          ] 226,53M  16,3MB/s   eta 10s    /dev/null                            63%[==========================================>                          ] 230,29M  16,3MB/s   eta 8s     /dev/null                            64%[===========================================>                         ] 234,36M  16,4MB/s   eta 8s     /dev/null                            65%[============================================>                        ] 237,98M  16,4MB/s   eta 8s     /dev/null                            66%[=============================================>                       ] 242,27M  16,4MB/s   eta 8s     /dev/null                            67%[=============================================>                       ] 245,98M  16,5MB/s   eta 8s     /dev/null                            68%[==============================================>                      ] 249,55M  16,6MB/s   eta 7s     /dev/null                            69%[==============================================>                      ] 252,34M  16,2MB/s   eta 7s     /dev/null                            70%[===============================================>                     ] 255,13M  15,9MB/s   eta 7s     /dev/null                            71%[================================================>                    ] 258,13M  15,8MB/s   eta 7s     /dev/null                            71%[================================================>                    ] 261,00M  17,3MB/s   eta 7s     /dev/null                            73%[=================================================>                   ] 267,01M  18,1MB/s   eta 6s     /dev/null                            74%[==================================================>                  ] 270,97M  18,3MB/s   eta 6s     /dev/null                            75%[===================================================>                 ] 275,38M  18,5MB/s   eta 6s     /dev/null                            76%[====================================================>                ] 279,36M  18,5MB/s   eta 6s     /dev/null                            78%[====================================================>                ] 283,84M  18,8MB/s   eta 6s     /dev/null                            79%[=====================================================>               ] 287,92M  18,9MB/s   eta 5s     /dev/null                            80%[======================================================>              ] 291,92M  18,9MB/s   eta 5s     /dev/null                            81%[=======================================================>             ] 296,75M  19,2MB/s   eta 5s     /dev/null                            82%[========================================================>            ] 300,91M  19,3MB/s   eta 5s     /dev/null                            83%[========================================================>            ] 301,58M  18,2MB/s   eta 5s     /dev/null                            83%[========================================================>            ] 304,01M  17,6MB/s   eta 4s     /dev/null                            84%[=========================================================>           ] 308,38M  17,9MB/s   eta 4s     /dev/null                            86%[==========================================================>          ] 313,25M  18,6MB/s   eta 4s     /dev/null                            87%[===========================================================>         ] 318,13M  19,2MB/s   eta 4s     /dev/null                            89%[============================================================>        ] 323,21M  19,7MB/s   eta 4s     /dev/null                            90%[=============================================================>       ] 328,00M  20,3MB/s   eta 2s     /dev/null                            91%[==============================================================>      ] 333,50M  20,1MB/s   eta 2s     /dev/null                            93%[===============================================================>     ] 338,54M  20,4MB/s   eta 2s     /dev/null                            94%[================================================================>    ] 344,15M  20,7MB/s   eta 2s     /dev/null                            96%[=================================================================>   ] 349,45M  21,1MB/s   eta 2s     /dev/null                            97%[==================================================================>  ] 355,19M  21,4MB/s   eta 0s     /dev/null                            99%[===================================================================> ] 360,63M  21,8MB/s   eta 0s     /dev/null                           100%[====================================================================>] 363,09M  21,9MB/s   in 21s      

2016-08-02 14:46:13 (17,3 MB/s) - ‘/dev/null’ saved [380731035/380731035]

--2016-08-02 14:46:13--  https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
Resolving upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)... 2620:0:862:ed1a::2:b
Connecting to upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)|2620:0:862:ed1a::2:b|:443... connected.
The certificate's owner does not match hostname ‘upload-lb.esams.wikimedia.org’
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 02 Aug 2016 11:46:13 GMT
  Content-Type: image/vnd.djvu
  Content-Length: 380731035
  Connection: keep-alive
  X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
  Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
  Etag: 026b82620912b415d1fdccfd0282c542
  X-Timestamp: 1424974981.50359
  X-Trans-Id: txd45aff4ab3e840d0bbbbd-0057a08784
  X-Varnish: 2230380404 2230114928, 3148478089 3148406992, 2055024997
  Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
  Accept-Ranges: bytes
  Age: 129
  X-Cache: cp1074 hit/3, cp3045 hit/1, cp3046 pass
  Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  X-Analytics: https=1;nocookies=1
  X-Client-IP: 2001:1bc8:1004::1
  Access-Control-Allow-Origin: *
  Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
  Timing-Allow-Origin: *
Length: 380731035 (363M) [image/vnd.djvu]
Saving to: ‘/dev/null’

/dev/null                             0%[                                                                     ]   1,21M  6,03MB/s              /dev/null                             3%[=>                                                                   ]  11,75M  29,4MB/s              /dev/null                             6%[===>                                                                 ]  24,42M  40,7MB/s              /dev/null                            10%[======>                                                              ]  39,52M  49,4MB/s              /dev/null                            15%[=========>                                                           ]  56,05M  56,0MB/s              /dev/null                            20%[=============>                                                       ]  74,01M  61,6MB/s              /dev/null                            25%[================>                                                    ]  92,04M  65,7MB/s              /dev/null                            30%[===================>                                                 ] 110,17M  68,8MB/s              /dev/null                            35%[=======================>                                             ] 128,35M  71,2MB/s              /dev/null                            40%[==========================>                                          ] 146,46M  73,2MB/s              /dev/null                            45%[==============================>                                      ] 164,59M  74,7MB/s              /dev/null                            50%[=================================>                                   ] 182,79M  76,1MB/s              /dev/null                            55%[=====================================>                               ] 200,73M  77,1MB/s              /dev/null                            60%[========================================>                            ] 218,78M  78,1MB/s              /dev/null                            65%[============================================>                        ] 236,92M  78,9MB/s   eta 2s     /dev/null                            70%[===============================================>                     ] 254,97M  83,7MB/s   eta 2s     /dev/null                            75%[==================================================>                  ] 272,84M  86,3MB/s   eta 2s     /dev/null                            80%[======================================================>              ] 290,70M  87,6MB/s   eta 2s     /dev/null                            85%[=========================================================>           ] 308,71M  89,6MB/s   eta 2s     /dev/null                            89%[=============================================================>       ] 326,66M  90,1MB/s   eta 0s     /dev/null                            94%[================================================================>    ] 344,57M  90,1MB/s   eta 0s     /dev/null                            99%[===================================================================> ] 362,11M  90,0MB/s   eta 0s     /dev/null                           100%[====================================================================>] 363,09M  90,0MB/s   in 4,4s    

2016-08-02 14:46:17 (82,3 MB/s) - ‘/dev/null’ saved [380731035/380731035]

So you're currently seeing 4+ MB/s, or ~40+ Mbps from eqiad, downloading from busy servers with many many connections/clients. That seems pretty respectable given the long distance across the Internet.

Yes, but of course Helsinki is a very happy place. Also, upload.wm.o is much faster than dumps.wm.o, which AFAICT never ever goes faster than 2 MB/s; both today are fast enough to saturate my home bandwidth, but I'll have to test more realistic conditions another day (e.g. university network and one of the typical 100+ Mbit/s connections nowadays provided in Milan).

upload is very different than dumps. It's part of our traffic-optimized cache cluster termination, and dumps isn't. Have you looked into optimization problems on dumps, re: sendfile/aio/directio, fs-cache, etc?

Oh, also if you're in Helsinki, hopefully you're hitting the upload.wm.o terminator in esams, which is close to you on the network. Dumps exists only in eqiad, which is much further away.

Tried another:

P3628 Bandwidth test from 151.0.247.226 for T120425

1	francesca@faggio:~$ for c in eqiad esams; do for p in 4 6; do wget -S -O /dev/null -$p --no-check-certificate --header='Host: upload.wikimedia.org' https://upload-lb.$c.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu ; done ; done
2	--2016-08-03 10:16:53-- https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
3	Risoluzione di upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)... 208.80.154.240
4	Connessione a upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)\|208.80.154.240\|:443... connesso.
5	AVVERTIMENTO: nessuno dei nomi alternativi indicati nel certificato corresponde al
6	nome dell'host richiesto "upload-lb.eqiad.wikimedia.org".
7	Richiesta HTTP inviata, in attesa di risposta...
8	HTTP/1.1 200 OK
9	Date: Wed, 03 Aug 2016 08:16:57 GMT
10	Content-Type: image/vnd.djvu
11	Content-Length: 380731035
12	Connection: keep-alive
13	X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
14	Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
15	Etag: 026b82620912b415d1fdccfd0282c542
16	X-Timestamp: 1424974981.50359
17	X-Trans-Id: txd45aff4ab3e840d0bbbbd-0057a08784
18	X-Varnish: 2426281118 2230114928, 188628845
19	Via: 1.1 varnish, 1.1 varnish
20	Accept-Ranges: bytes
21	Age: 73973
22	X-Cache: cp1074 hit/4, cp1064 pass
23	Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
24	X-Analytics: https=1;nocookies=1
25	X-Client-IP: 151.0.247.226
26	Access-Control-Allow-Origin: *
27	Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
28	Timing-Allow-Origin: *
29	Lunghezza: 380731035 (363M) [image/vnd.djvu]
30	Salvataggio in: "/dev/null"
31
32	/dev/null 100%[===================================================================>] 363,09M 3,12MB/s in 2m 55s
33
34	2016-08-03 10:19:52 (2,08 MB/s) - "/dev/null" salvato [380731035/380731035]
35
36	--2016-08-03 10:19:52-- https://upload-lb.eqiad.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
37	Risoluzione di upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)... 2620:0:861:ed1a::2:b
38	Connessione a upload-lb.eqiad.wikimedia.org (upload-lb.eqiad.wikimedia.org)\|2620:0:861:ed1a::2:b\|:443... non riuscito: La rete non è raggiungibile.
39	--2016-08-03 10:19:52-- https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
40	Risoluzione di upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)... 91.198.174.208
41	Connessione a upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)\|91.198.174.208\|:443... connesso.
42	AVVERTIMENTO: nessuno dei nomi alternativi indicati nel certificato corresponde al
43	nome dell'host richiesto "upload-lb.esams.wikimedia.org".
44	Richiesta HTTP inviata, in attesa di risposta...
45	HTTP/1.1 200 OK
46	Date: Wed, 03 Aug 2016 08:19:52 GMT
47	Content-Type: image/vnd.djvu
48	Content-Length: 380731035
49	Connection: keep-alive
50	X-Object-Meta-Sha1base36: cgtju44ck09715vjyz90cs3qphwkxm0
51	Last-Modified: Thu, 26 Feb 2015 18:23:02 GMT
52	Etag: 026b82620912b415d1fdccfd0282c542
53	X-Timestamp: 1424974981.50359
54	X-Trans-Id: txd45aff4ab3e840d0bbbbd-0057a08784
55	X-Varnish: 2230380404 2230114928, 3345877660 3148406992, 3253789253
56	Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
57	Accept-Ranges: bytes
58	Age: 74149
59	X-Cache: cp1074 hit/3, cp3045 hit/3, cp3034 pass
60	Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
61	X-Analytics: https=1;nocookies=1
62	X-Client-IP: 151.0.247.226
63	Access-Control-Allow-Origin: *
64	Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
65	Timing-Allow-Origin: *
66	Lunghezza: 380731035 (363M) [image/vnd.djvu]
67	Salvataggio in: "/dev/null"
68
69	/dev/null 100%[===================================================================>] 363,09M 3,93MB/s in 1m 41s
70
71	2016-08-03 10:21:33 (3,61 MB/s) - "/dev/null" salvato [380731035/380731035]
72
73	--2016-08-03 10:21:33-- https://upload-lb.esams.wikimedia.org/wikipedia/commons/9/98/J%C3%A6rnet.djvu
74	Risoluzione di upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)... 2620:0:862:ed1a::2:b
75	Connessione a upload-lb.esams.wikimedia.org (upload-lb.esams.wikimedia.org)\|2620:0:862:ed1a::2:b\|:443... non riuscito: La rete non è raggiungibile.

• ayounsi moved this task from Backlog to Next quarter on the netops board.Jun 27 2017, 2:49 PM

@Nemo_bis I see that the last comment is from more than a year ago, is that issue still happening?

In T120425#3763986, @ayounsi wrote:

@Nemo_bis I see that the last comment is from more than a year ago, is that issue still happening?

Setting status to stalled per last comment. @Nemo_bis: After you have provided the information asked for and if this still happens, please reset the status of this report back to "Open" via the Add Action... → Change Status dropdown. Thanks!

Unfortunately closing this report as no further information has been provided and as the last updates in this task happened 21 months ago.
@Nemo_bis: After you have provided the information asked for and if this still happens, please set the status of this report back to "Open" via the Add Action... → Change Status dropdown. Thanks!

ArielGlenn moved this task from Backlog to Done on the Datasets-General-or-Unknown board.Jul 2 2018, 12:38 PM

dumps.wikimedia.org seems to have poor throughput towards some destinationsClosed, DeclinedPublicActions