How can we provide good bandwidth for downloading to all comers without connection/bw caps and without impacting dumps performance? In theory this should be separate for the desire for mirrors of our datasets.
|Open||None||T128514 Dumps 2.0 Performance design questions|
|Open||None||T128874 How can downloaders get good bandwidth with no impact on dumps production?|
We've been using mirrors to download wikibase dumps for populating WDQS because using our own endpoints is ratelimited, is this task is also about allowing to more efficiently use our own dump internally or just for external downloaders?
This task is for the (someday over the rainbow...) dumps rewrite/rearchitecture, on hold for several years now because we have no resources to allocate to it. The dumps are available via nfs to stat100x, I forget which instances. Can something like that work for you? I guess we should have a new task for this if not. Or we can take the topic to T191491