Page MenuHomePhabricator

download.wikimedia.org is slow from Telecom Italia
Closed, DuplicatePublic

Description

My average download speed from download.wikimedia.org is 200/250kb/s, this night it's about 50kb/s, ranging from some kb to 100kb/s. Dunno if it's a Telecom Italia Sparkle/Seabone (the carrier my ISP uses) problem or WMF's one but it's maybe worth further investigation.

Event Timeline

Vituzzu raised the priority of this task from to Needs Triage.
Vituzzu updated the task description. (Show Details)
Vituzzu subscribed.

Yep! Yesterday or maybe the day before some thumbsnails (namely old file versions in Commons image history) didn't load.

for completeness because we talked about it on IRC:

Recently on Aug 25 with change 233659 download.wm has been switched to point to the general text cluster and get redirected to dumps.

Before that it used to be a straight CNAME for dumps which is dataset1001, so it didn't go via the cluster.

This was to fix T107575 the SSL cert issue but is probably not related.

@Vituzzu Yesterday/today in the morning there was network saturation on the dumps host, I think due to nginx being saturated by requests/bots. This calmed later today. Do you continue suffering the same issue right now?

faidon triaged this task as High priority.Sep 11 2015, 4:34 PM
faidon edited projects, added Datasets-General-or-Unknown; removed netops.
faidon added a subscriber: ArielGlenn.
faidon subscribed.

We got alerts from both Watchmouse and Catchpoint about dumps yesterday. It doesn't look networking-related but more like dumps-related. Adjusting the tasks' metadata accordingly.

@jcrespo it took about 6:30 hours for a 2.6gb dump (roughly 120kb/s avg speed then) which is significantly slower than usual but definitely faster than at the time I opened this task. Now I ran a test by downloading the first 100mb of the same dump with a speed ranging from 100 to 650kb/s (I'm on a g.dmt ADSL, so this is pretty fast) 350 kb/s avg.

@faidon sorry for being inexact on my wording (I didn't add the netops tag), with network saturation, I really meant too many requests at application level for the dumps host (server-related, not physical network-related).

I got the timeouts, but they were general connection slowness on the nginx, and the service never had an outage (just become too slow, as Vituzzu).

I can only see two ways to solve this long term (I think there is no longer an issue now): increase the provisioned hardware (difficult) or introduce QoS/more strict throttling (in addition to the one it is already in place).

Then it seems more like a duplicate of T45647 ...