Change Details

We would like to host Zim files on WMF infrastructure for the upcoming Android compilations feature. On average, these files will vary in size from 1 GB - 20 GB. Clients will use HTTP to download and will need support for byte offsets and ranges… and/or chunking. 1. Where can we host these files? 2. What caching considerations do we need to make? 3. What about CDNs? 4. Do we host any existing large files that we can use as a template? So far we have 2 candidates for where to host: 1. Swift (along with Commons media) 2. Dumps For example, Kiwix hosts it's Zim files here: http://wiki.kiwix.org/wiki/Content_in_all_languages

We would like to host Zim files on WMF infrastructure for the upcoming Android compilations feature. On average, these files will vary in size from 1 GB - 20 GB. Clients will use HTTP to download and will need support for byte offsets and ranges… and/or chunking. 1. Where can we host these files? 2. What caching considerations do we need to make? 3. What about CDNs? 4. Do we host any existing large files that we can use as a template? So far we have 2 candidates for where to host: 1. Swift (along with Commons media) 2. Dumps For example, Kiwix hosts its Zim files on their own server, with various academic and private mirrors in Europe and other continents; MirrorBrain directs every download to a suitable [[https://github.com/kiwix/maintenance/blob/1f7a7d4fb496e9fa49ddcc47f8b632b297bacafe/maintenance_tools/update_mirrorbrain_db.sh#L35-L63|mirror]] (the dumps host serves only a negligible part of the total bandwidth): http://wiki.kiwix.org/wiki/Content_in_all_languages

We would like to host Zim files on WMF infrastructure for the upcoming Android compilations feature. On average, these files will vary in size from 1 GB - 20 GB. Clients will use HTTP to download and will need support for byte offsets and ranges… and/or chunking. 1. Where can we host these files? 2. What caching considerations do we need to make? 3. What about CDNs? 4. Do we host any existing large files that we can use as a template? So far we have 2 candidates for where to host: 1. Swift (along with Commons media) 2. Dumps For example,For example, Kiwix hosts its Zim files on their own server, with various academic and private mirrors in Europe and other continents; Kiwix hosts it's Zim files hereMirrorBrain directs every download to a suitable [[https://github.com/kiwix/maintenance/blob/1f7a7d4fb496e9fa49ddcc47f8b632b297bacafe/maintenance_tools/update_mirrorbrain_db.sh#L35-L63|mirror]] (the dumps host serves only a negligible part of the total bandwidth): http://wiki.kiwix.org/wiki/Content_in_all_languages