Page MenuHomePhabricator

Request creation of dumpstorrents VPS project
Closed, ResolvedPublic

Description

Project Name: torrents

Developer account usernames of requestors: samtar

Purpose: A continuation of the idea started by @Legoktm (https://dump-torrents.toolforge.org) — automatically create web-seeded torrent files from completed data dumps.

Brief description: Using mktorrent (GPLv2), create a torrent file from newly dumped database dumps. This will require the /public/dumps NFS mount. The generated torrents will use web seeds to ensure they don't suffer too badly from lack of seeders. In an ideal world, we'd also host a restricted torrent tracker (perhaps this, MIT) instead of relying on other trackers (such as fosstorrents), but that's probably a question for another time.
See also T354174: Automatically generate torrent file and T29653: Provide dumps using bittorrent

How soon you are hoping this can be fulfilled: This month (though, no rush)

Event Timeline

This sounds mostly fine, although I have a couple of concerns:

  1. Can you pick a more specific name that specifies what is being torrented? Like maybe dumptorrents or similar?
  1. I don't know a ton about how torrents work, but will this drastically increase load on NFS other torrent peers sync with our copies? Or...
  1. If you're planing to cache dumps locally to avoid NFS load, do you know how much space that will require?

+1 (pending on Andrew's questions)

Thanks for the comments @Andrew

  • dumpstorrents sounds like a better name, agreed
  • the concerns raised here suggest that it does increase NFS load, though perhaps not to a disruptive level, given how 'infrequently' new torrents would need to be generated? but,
  • if cached locally, enough space to hold a few of the most recent dump/categories (300GB?) while the torrent is being generated
Aklapper renamed this task from Request creation of torrents VPS project to Request creation of dumpstorrents VPS project.Jul 1 2025, 12:28 PM
  • the concerns raised here suggest that it does increase NFS load, though perhaps not to a disruptive level, given how 'infrequently' new torrents would need to be generated?

Bursts are the main concern here, and it sounds like you're describing something bursty! Is this something with a throttle you can configure?

  • the concerns raised here suggest that it does increase NFS load, though perhaps not to a disruptive level, given how 'infrequently' new torrents would need to be generated?

Bursts are the main concern here, and it sounds like you're describing something bursty! Is this something with a throttle you can configure?

mktorrent itself doesn't seem to have a way of throttling this, so initially I think I'll:

  1. copy a dump from the dumps mount to the 'local disk'
  2. run mktorrent on that
  3. delete the copied dump
  4. repeat

That sounds like it'll be less 'bursty' :)

Andrew claimed this task.