Page MenuHomePhabricator

Automatically generate torrent file
Closed, DuplicatePublicFeature

Description

As someone who creates & seeds data dump torrents, I would like a torrent file to be created as part of the dumping process, so that I don't need to manually create one, and more than the few listed here are available for seeding.

I would suggest that initially these rely on web seeds, so that WMF don't need to immediately worry about seeding these mass-created torrent files.

This could be achieved using mktorrent — I will note that most torrents are created within a few seconds, but large torrents (e.g. enwiki's pages-articles-multistream.xml.bz2) can take a few minutes.


I use the very hacky python snippet below to create new torrents:

niceness = "19"
filter_type = 'pages-articles-multistream.xml.bz2'
filter_date = '20231220'
project = 'enwiki'

create_command = f"nice --adjustment={niceness} /usr/bin/mktorrent -a http://tracker.opentrackr.org:1337/announce -a udp://tracker.opentrackr.org:1337 -a udp://tracker.openbittorrent.com:80/announce -a http://fosstorrents.com:6969/announce -a udp://fosstorrents.com:6969/announce -w https://dumps.wikimedia.org/{project}/{filter_date}/{project}-{filter_date}-{filter_type} -w http://dumps.wikimedia.your.org/{project}/{filter_date}/{project}-{filter_date}-{filter_type} -w http://ftp.acc.umu.se/mirror/wikimedia.org/dumps/{project}/{filter_date}/{project}-{filter_date}-{filter_type} -w http://wikipedia.c3sl.ufpr.br/{project}/{filter_date}/{project}-{filter_date}-{filter_type} -o {project}-{filter_date}-{filter_type}.torrent -l 20 /public/dumps/public/{project}/{filter_date}/{project}-{filter_date}-{filter_type}"

subprocess.run(base, shell=True)