MZMcBride asked me to test it, so using wget my average speed was 47.3 KB/s and it took a total of 6m 18s.

<carl-m> I am getting really slow download speeds from dumps.wikimedia.org to the toolserver - slow like 68KB/sec . Is that expected?

Right now it's very fast, it saturates my home connection and is about 10 MiB/s on Toolserver: lowering priority.
At the time this bug was filed, and for several days on, people from several different places had super-slow download as reported; Ariel didn't see any obvious network problem, and there were people happily downloading at 5 MiB/s.
We also suspected the sitenotice on es.wiki in early January making people download Kiwix's ZIM files mirrored on dataset2, but the network graphs don't show any big peak (plus, Kelson comments that his personal server has about the same traffic, just for Kiwix): http://ganglia.wikimedia.org/latest/?r=year&cs=&ce=&c=Miscellaneous+pmtpa&h=dataset2.wikimedia.org&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS
However, at the moment CPU nice is 6 times the yearly average (7,92 vs. 1,33) and CPU wio 3 (9,92 vs. 3,61).

Danny_B> dumps site is slow... downloading small rss file takes 10 secs... :-/

Network activity for dataset2 only 30 MB/s now vs. 50 average, download of a big files starts at some 10 KB/s speed for a while and eventually is something like 2-300 on Toolserver.

(In reply to comment #3)

However, at the moment CPU nice is 6 times the yearly average (7,92 vs. 1,33)
and CPU wio 3 (9,92 vs. 3,61).

This seems to vary with little effect on download speed.
The only other change I can see in ganglia is that, about the time this bug was filed, ~2 TB were freed on disk. http://ganglia.wikimedia.org/latest/graph.php?c=Miscellaneous%20pmtpa&h=dataset2.wikimedia.org&v=7317.328&m=disk_free&r=year&z=default&jr=&js=&st=1361113707&vl=GB&ti=Disk%20Space%20Available

From the weekly graph it would seem that something managed to pull 60 MB/s about a week ago, which matches the observation that while download is very slow for most people there are still some who manage to download very very quickly.

After we moved to the nginx web server on the dumps host, I let bandwidth and cap limits expire. Sure enough, some folks, unwittingly or not, took advantage of that so I reinstated them but was too agressive about the bw limits. You should see better behavior now. The caps are a little lower than they were but the connection limits are up to 3 per ip from the previous 2.

I still get ~100KB/sec with the current limits. At that speed, downloading an hours worth of views takes ~15 minutes, so downloading a day is thus ~6 hours. It's normally not a problem, since it's done fairly continuously, but catching up stats.grok.se when the collection has failed for a week will now take a few days, which is perhaps a bit unfortunate - I'm getting requests. :)

can you go with three connections all downloading and see how that is? Or is that with multiple connections?

No, about 100KB/sec is with a single connection. I can code up a parallel downloader to try, but fortunately there's no pressing need at the moment: it's caught up now.

Now, with three parallel downloads, the speed is about 1.3 MB/s for each of them, from a server in Finland.

Disk I/O is saturated:

root@dataset1001:~# iostat -h -m -t -x -d 5 dm-0
Linux 3.2.0-75-generic (dataset1001) 	02/16/2015 	_x86_64_	(4 CPU)

02/16/2015 09:33:01 PM
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
dm-0
                  0.00     0.00  326.21   65.89    35.84     3.57   205.85     2.94    7.50    6.52   12.40   1.75  68.63

02/16/2015 09:33:06 PM
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
dm-0
                  0.00     0.00   45.60  708.40     5.49    87.69   253.08   165.71  226.03  475.91  209.95   1.33 100.00

02/16/2015 09:33:11 PM
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
dm-0
                  0.00     0.00   47.00 1231.60     5.49   152.35   252.81   167.46  128.40  398.01  118.11   0.78 100.00

In T45647#1042316, @ori wrote:

Disk I/O is saturated:

I guess that explains the erratic speeds I saw to a server in the UK - from over 2MB/s to barely a few KB/s

reedy@ko-kra:~$ wget https://dumps.wikimedia.org/other/pagecounts-raw/2015/2015-02/pagecounts-20150210-170000.gz
--2015-02-16 21:25:46--  https://dumps.wikimedia.org/other/pagecounts-raw/2015/2015-02/pagecounts-20150210-170000.gz
Resolving dumps.wikimedia.org (dumps.wikimedia.org)... 2620:0:861:1:208:80:154:11, 208.80.154.11
Connecting to dumps.wikimedia.org (dumps.wikimedia.org)|2620:0:861:1:208:80:154:11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 116432248 (111M) [application/octet-stream]
Saving to: ‘pagecounts-20150210-170000.gz’

100%[==============================================================================================================================>] 116,432,248  850KB/s   in 2m 18s

2015-02-16 21:28:10 (823 KB/s) - ‘pagecounts-20150210-170000.gz’ saved [116432248/116432248]

reedy@ko-kra:~$

In T45647#1042316, @ori wrote:

Disk I/O is saturated:

This is known; this is the reason bandwidth & connection limits were instituted (this was troubleshot on IRC and sadly, not copied here). It was supposed to be a hotfix until we get more resources for it, although I don't remember seeing any hardware request for it. Maybe @ArielGlenn has something?

The speed of downloads from dumps varies considerably depending on the *file* you download, probably due to different disks involved. If some disk/partition happened to be less busy than others, some directory reshuffling could be an easy win.

(This week is probably not very representative because a big sync is ongoing.)

It's an lvm on hardware raid 6 so files should be spread about across disks. We don't have multiple filesystems/partitions so there's no shuffling around of files to be done.

After scrying through the access logs to check traffic patterns, looking at the behavior before we went to a 10G nic on dataset1001, and staring some more at iotop, lsof and etc., I think something we could do quickly that should help alot would be to add ms1001.wikimedia.org in as a host that responds to dumps/download.wm.org via rr or so. Its rsync is a bit behind so I'd need to fix that up, make sure that it gets an ipv6 address and so on, It's got comparable disks but more of them, more memory than dataset1001 and more cores too.

Nemo, what sort of variance are you seeing on file downloads? If you try the same file a few times in a row do you see a change then as well or no?

Note that the kiwix rsync finished up but the ms1001 rsync will throw things off again, though it, like the kiwix rsync, is bw capped.

The ms1001 rsync should kick off in about an hour, I'll give an ETA of finishing time about an hour after that. Not more than a couple of days I would think.

Nemo, what sort of variance are you seeing on file downloads?

When jhs reported the Wikidata XML dump was very slow, I tried downloading that and other XML dumps; I remember that day the Wikidata XML dump was three times slower than other (latest) XML dumps of similar size. It didn't matter whether I started the downloads in parallel or sequentially, IIRC, but I didn't rigorously test.

Could it be something about a file being accessed by multiple connections at the same time?

Forgot to give an ETA. But the rsync just finished. We're back to regular rsyncs out of cron now. Waiting for bond of the ethernet interfaces on the host, need to figure out the best way to pool it with dataset1001 into a cluster of 2, then we'll see how network traffic is at that point.

Nemo, that's pretty weird and I can't think of a good reason we'd see that behavior now and not in the past, as it's always been the case that when certain files are available the hordes come to download at the same time. I'm going to put off looking into it until we have a second machine serving files and we'll see how that does.

It would be nice to have those nginx stats for dumps hosts: https://ganglia.wikimedia.org/latest/graph_all_periods.php?title=&vl=&x=&n=&hreg%5B%5D=.*&mreg%5B%5D=.*nginx.*&gtype=line&glegend=show&aggregate=1

Jeff, can you copy the relevant module from frack puppet to operations/puppet, please?

Note the tech press recently ran some articles (e.g. https://thestack.com/cloud/2015/09/09/wikipedia-anne-hathaway-open-source-web-trends-japan-research/ ) about this research paper: http://arxiv.org/pdf/1509.02218v1.pdf . The paper itself cites our pageview data on dumps.wm.o as the source they're looking at, and encouraging others to look at because we're offering some pretty unique timely and information that commercial entities find desirable. I wouldn't be surprised if this lead to a recent traffic increase on dumps.

I wouldn't be surprised if this lead to a recent traffic increase on dumps.

Traffic doesn't look exceptional though, around 100 MB/s. There were 4 other similar peaks in the last 7 months https://ganglia.wikimedia.org/latest/?r=month&cs=02%2F01%2F2015+00%3A00&ce=09%2F30%2F2015+00%3A00&m=cpu_report&c=Miscellaneous+eqiad&h=dataset1001.wikimedia.org&tab=m&vn=&hide-hf=false&mc=2&z=medium&metric_group=NOGROUPS

P.s. The new summary is unproven. Most of the times, when we get such reports, there is no peak in usage at all. Moreover, we often experience that only some files are slow, not others, so the total usage of the machine can't be at fault.

Please note than on the original ticket, I noted that I do not think this is a network bandwith problem at all. I think, however, that there is some kind of contention on the server. The timouts are real and the semi-outage is real as detected by our own monitoring. I cannot demonstrate, however, that it is caused by too many requests or other problem (the contention could make the accesses being actually lower than average). These are the hourly statistics of access in the last days:

The CPU wait time went up to 40% and the load went up to 10- I think we are not serving requests as fast as we can. At those points, network speed is lower than normally, if any: https://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&m=cpu_report&c=Miscellaneous+eqiad&h=dataset1001.wikimedia.org&tab=m&vn=&hide-hf=false&mc=2&z=medium&metric_group=NOGROUPS_|_cpu_|_disk_|_load_|_network_|_process

I'll see if I can correlate the times to server activity to get a lead on this.

I did not see anything strange happening this month's run, which has now concluded. Can folks on this task let me know if there was a period during which they saw a real slowdown, and if so, when?

Reminder that the monthly fulls are running again; if people could keep an eye out and let me know particular intervals when they have issues, that would help a lot. Thanks!

Worksforme!

I've yet to find a file or occasion where average download speed goes over 1.5-2 MB/s... rather painful when trying to do ad hoc analysis.

Is someone still suffering from this issue anymore? If not, it should be closed.