We had bandwidth and number of connections severely capped on dataset1001; there was also a memory setting for dealing with those heavy rsync reads. We should revisit all of those, now that we are on a beefier server.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | Hydriz | T101639 Make use of Wikidata JSON dumps in /public/dumps | |||
| Open | None | T191491 Adjust bandwidth/connection limits, memory settings on labstore1006,7 as appropriate |
A brief check suggests not many changes are a good idea. It needs some more work, but I'm not working on it right now, so I am tossing it back on the stack.
Will it be possible to lift the bandwidth cap for specific hosts within CloudVPS? Currently, the limit is at 5 MB/s which is painfully slow when it comes to archiving the dumps, Wikibase dumps alone is at 560 GB, it will take almost the whole week to archive just one item, not forgetting that there are also other dumps that will take up the bandwidth as well.
Hey @nskaggs I figure this task has been around long enough it may have gotten lost in the shuffle. Of course the right hosts are now clouddumps1001,2. Any thoughts?
@ArielGlenn Yes, there are much different machines present now, but I wonder if the networking backend has also improved? Is there a way to test out these changes? It sounds like you have an idea of what changes to make. How can I help?
I don't know, since I haven't been involved with these or similar hosts since WMCS took them over! But the question is wheher the rsync bandwidth cap can be loosened up somewhat without a negative impact on existing services.