Page MenuHomePhabricator
Create Task
Maniphest T172123

Determine how to upload Zim files to Swift infrastructure
Closed, InvalidPublic
Actions

Description

Swift appears to be a candidate for hosting Zim files. Several steps need to be accomplished to validate this assumption:

  1. Figure out how to upload from the Cloud VPS instance where the Zim files will be created
  2. Figure out how to chunk the files so they can be hosted efficiently
  3. Upload several test files which are representative of the Zim files that would be hosted
  4. Test downloading the files to the Android app

For the test files we should generate and upload the following for the en project (Based on this sheet):

  1. Wiki Medicine
  2. Core Wikipedia
  3. 5000 Most Read
  4. 50000 Most Read

Each of those collections should be generated with:

  1. All content
  2. No audio or video
  3. No images, audio or video

This is a total of 12 collections

Some references:
https://wikitech.wikimedia.org/wiki/Swift/How_To
https://wikitech.wikimedia.org/wiki/Media_storage
https://wikitech.wikimedia.org/wiki/Swift

Details

SubjectRepoBranchLines +/-
hieradata: create pagecompilation accountoperations/puppetproduction+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Fjalapeno created this task.Jul 31 2017, 3:25 PM
• Fjalapeno updated the task description. (Show Details)
• Fjalapeno moved this task from Kanban to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.
• Fjalapeno edited projects, added Product-Infrastructure-Team-Backlog-Deprecated; removed Product-Infrastructure-Team-Backlog-Deprecated (Kanban).
• Fjalapeno updated the task description. (Show Details)Jul 31 2017, 3:28 PM
fgiunchedi added a comment.Aug 1 2017, 9:01 AM

I'll add some thoughts/braindump below:

  1. For production swift access isn't permitted from cloud vps, does the zim generation and upload need to happen in cloud vps?
  1. The simplest way for chunking I believe could be via swift's "dynamic large objects", essentially upload the file chunks with specific filenames (in the same swift container) so that swift can reassemble the chunks correctly. (TODO, verify e.g. range requests work as expected) See also https://docs.openstack.org/swift/latest/overview_large_objects.html#module-swift.common.middleware.dlo

Since for DLO the chunks need to be in the same container we could shard the containers e.g. 256 ways based on the filename hash (like it happens in production for big wikis and commons).
e.g. using the swift commandline client and a file with hash 4b712aeabedb0109b1099b0b5fe34508: swift upload zim_4b -S 1073741824 4b712aeabedb0109b1099b0b5fe34508 and the file would be available for download at https://swift_hostname/zim_4b/4b712aeabedb0109b1099b0b5fe34508

  1. To test download/upload in beta let me know a swift username to be used and I'll create the account in beta and production
fgiunchedi mentioned this in T172148: Determine URL paths for Zim files.Aug 1 2017, 9:10 AM
• ema moved this task from Backlog to Caching on the Traffic board.Aug 1 2017, 12:44 PM
• Mholloway added a comment.EditedAug 1 2017, 7:16 PM

Hey @fgiunchedi,

  1. For production swift access isn't permitted from cloud vps, does the zim generation and upload need to happen in cloud vps?

In principle it doesn't have to happen in Cloud VPS, that's just where I'm prototyping right now, though we were thinking (or hoping) it would be an appropriate place for an occasional (~1x/month) zim generation job, if possible to just keep it running there after the prototyping is finished. If that's out of the question, then we'd have to talk to someone about finding some appropriate production cluster hardware...

I shouldn't have any trouble accessing the beta cluster Swift instance from Cloud VPS, correct?

  1. The simplest way for chunking I believe could be via swift's "dynamic large objects", essentially upload the file chunks with specific filenames (in the same swift container) so that swift can reassemble the chunks correctly. (TODO, verify e.g. range requests work as expected) See also https://docs.openstack.org/swift/latest/overview_large_objects.html#module-swift.common.middleware.dlo

Since for DLO the chunks need to be in the same container we could shard the containers e.g. 256 ways based on the filename hash (like it happens in production for big wikis and commons).
e.g. using the swift commandline client and a file with hash 4b712aeabedb0109b1099b0b5fe34508: swift upload zim_4b -S 1073741824 4b712aeabedb0109b1099b0b5fe34508 and the file would be available for download at https://swift_hostname/zim_4b/4b712aeabedb0109b1099b0b5fe34508

Thanks! I'll try that approach first when I get to the uploading stage.

  1. To test download/upload in beta let me know a swift username to be used and I'll create the account in beta and production

Let's use the username mdholloway.

Thanks again!

• NHarateh_WMF moved this task from Needs Triage to Tracking on the Wikipedia-Android-App-Backlog board.Aug 2 2017, 3:18 PM
fgiunchedi added a comment.Aug 3 2017, 1:22 PM
In T172123#3490510, @Mholloway wrote:

Hey @fgiunchedi,

  1. For production swift access isn't permitted from cloud vps, does the zim generation and upload need to happen in cloud vps?

In principle it doesn't have to happen in Cloud VPS, that's just where I'm prototyping right now, though we were thinking (or hoping) it would be an appropriate place for an occasional (~1x/month) zim generation job, if possible to just keep it running there after the prototyping is finished. If that's out of the question, then we'd have to talk to someone about finding some appropriate production cluster hardware...

Yeah it shouldn't be hard to find a spot in the production cluster to run the generation job, running everything on a cloud vm for prototyping now is more than adeguate.

I shouldn't have any trouble accessing the beta cluster Swift instance from Cloud VPS, correct?

From inside beta it should be already accessible, from other projects I'm not sure but I'm assuming your vm runs in beta already?

  1. To test download/upload in beta let me know a swift username to be used and I'll create the account in beta and production

Let's use the username mdholloway.

I'd like to keep the username tied to the service rather than a person, maybe pagecompilation ? zimdumps ?

• Mholloway added a comment.EditedAug 3 2017, 1:49 PM
In T172123#3496939, @fgiunchedi wrote:
In T172123#3490510, @Mholloway wrote:

I shouldn't have any trouble accessing the beta cluster Swift instance from Cloud VPS, correct?

From inside beta it should be already accessible, from other projects I'm not sure but I'm assuming your vm runs in beta already?

Looks like the beta cluster is itself a project in Labs/Cloud VPS (is that what you mean by beta?) as well, so yes, I think we're on the same page.

  1. To test download/upload in beta let me know a swift username to be used and I'll create the account in beta and production

Let's use the username mdholloway.

I'd like to keep the username tied to the service rather than a person, maybe pagecompilation ? zimdumps ?

Good point. I like pagecompilation.

• Mholloway added a parent task: T172732: Create a production-ready zim content pack generation and upload service.Aug 7 2017, 7:47 PM
• Mholloway created subtask T172735: Create 'pagecompilation' Swift account(s) (beta + prod) for Readers offline article compilations project.
gerritbot subscribed.Aug 11 2017, 8:40 PM

Change 371579 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: create pagecompilation account

https://gerrit.wikimedia.org/r/371579

gerritbot added a project: Patch-For-Review.Aug 11 2017, 8:40 PM
• Mholloway mentioned this in T172732: Create a production-ready zim content pack generation and upload service.Aug 12 2017, 2:26 PM
• Mholloway closed subtask T172735: Create 'pagecompilation' Swift account(s) (beta + prod) for Readers offline article compilations project as Resolved.Aug 12 2017, 2:29 PM
gerritbot added a comment.Aug 16 2017, 1:43 PM

Change 371579 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: create pagecompilation account

https://gerrit.wikimedia.org/r/371579

• ema triaged this task as Medium priority.Sep 28 2017, 2:49 PM
• Mholloway moved this task from Tracking to Upcoming on the Product-Infrastructure-Team-Backlog-Deprecated board.Apr 6 2018, 8:06 PM
• Mholloway removed • Mholloway as the assignee of this task.Apr 12 2018, 2:48 PM
• Mholloway closed this task as Invalid.Jun 18 2018, 8:32 PM

This is stalled, possibly indefinitely. Consider reopening if and when this work picks back up.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL