I am using this server (wcdo) to create some datasets and run some data visualizations.
I need to download dumps regularly and I'd like to know if someone could grant access public dumps so I could skip the downloading part and speed up some processes. I've seen in this site that this is possible.
https://wikitech.wikimedia.org/wiki/Help:Shared_storage#/public/dumps
Thanks.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| cloudstore: mount dumps NFS on wcdo Cloud VPS project | operations/puppet | production | +4 -0 |
(Not sure which project tag to add here; https://wikitech.wikimedia.org/wiki/Help:Shared_storage would welcome a more specific link).
Change 524319 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloudstore: mount dumps NFS on wcdo Cloud VPS project
Change 524319 merged by Bstorm:
[operations/puppet@production] cloudstore: mount dumps NFS on wcdo Cloud VPS project
Hi,
I noticed that reading the dumps at /public/dumps from the wcdo.eqiad.wmflabs server is much slower now than the months after this NFS share was created.
Originally, I used to copy the dumps from the website by downloading them, and I needed a lot of space in disk to have the wikidata dump, for instance. After reading it, I could delete it, but I needed that space.
Now I cannot download them because I run out of space (it is occupied by the databases and the dump is also bigger). What could we do? Expand the disk? Or is there something to make the access to public as fast as it was before? I would need to have a solution because this blocks updating data for the website.
Thank you very much.
Please open a new bug report rather than making a comment on one that has been closed for many months. It would be helpful if in that report you could describe more precisely "much slower" and ideally describe your use case and how it is IOPS dependent.
Hi @Bstorm. Could you please create the same link for the server (chm)? The reason is the same. We need to access /public/dumps directly to avoid downloading it to the server and occupying space.
Thank you very much.
Best,
Marc
PS: I created this other task: https://phabricator.wikimedia.org/T279558