Page MenuHomePhabricator

How can downloaders get good bandwidth with no impact on dumps production?
Open, MediumPublic

Description

How can we provide good bandwidth for downloading to all comers without connection/bw caps and without impacting dumps performance? In theory this should be separate for the desire for mirrors of our datasets.

Event Timeline

We've been using mirrors to download wikibase dumps for populating WDQS because using our own endpoints is ratelimited, is this task is also about allowing to more efficiently use our own dump internally or just for external downloaders?

This task is for the (someday over the rainbow...) dumps rewrite/rearchitecture, on hold for several years now because we have no resources to allocate to it. The dumps are available via nfs to stat100x, I forget which instances. Can something like that work for you? I guess we should have a new task for this if not. Or we can take the topic to T191491

Thanks, I think there is a ticket T222349 (which is where I should have pinged in the first place)