Determine where to generate Zim content packs using Production WMF infrastructure
Closed, InvalidPublic
Actions

Assigned To

Authored By

	• Fjalapeno
	Jul 17 2017, 4:07 PM

Description

We would like to generate Zim files for use in WMF products. For this we are developing a Zim generation service that will:

Create Zims
Upload Zims to Swift
Track the metadata/location of Zims in a database in order to share with other services.

For a first step, we would like to create a few files to get a handle on the problem. This will be done in a Cloud VPS instance and then uploaded to the Beta Swift instance.

Once we have done this and gathered some data, we want to move the generation service to another place in Production hardware. The hardware has a few requirements:

Run long running jobs (on the order of hours) in order to generate the ZIms
Access the production Swift instance in order to upload and manage files
Host a database or have access to a database that it can insert metadata into. This database MUST be accessible by the Mobile Content Service in SCB
Make a request to an internal API in the Mobile Content Service

Related Objects
Search...

Status	Assigned	Task
Declined	Dbrant	T163706 [Spike] [4hrs] Investigate loading a random page from a zim file using an interceptor
Resolved	RHo	T166599 [EPIC] Create the user interface flow for readers to use offline content packs
Resolved	• Mholloway	T163714 Create a page content pack info client
Invalid	None	T172732 Create a production-ready zim content pack generation and upload service
Invalid	• Fjalapeno	T170841 Determine where to generate Zim content packs using Production WMF infrastructure
Resolved	• Mholloway	T172296 Determine how to gather top-viewed article lists for use in generating ZIM files
Invalid	• Fjalapeno	T172734 Collect data on zim file generation and upload times

Event Timeline

• Fjalapeno created this task.Jul 17 2017, 4:07 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 17 2017, 4:07 PM

• Fjalapeno added a project: Android-app-feature-Compilations.Jul 17 2017, 4:07 PM

• Fjalapeno moved this task from Needs triage to Backlog on the Product-Infrastructure-Team-Backlog-Deprecated board.Jul 17 2017, 4:14 PM

• NHarateh_WMF moved this task from Needs Triage to Tracking on the Wikipedia-Android-App-Backlog board.Jul 19 2017, 3:30 PM

• Fjalapeno added a parent task: T163714: Create a page content pack info client.Jul 19 2017, 4:23 PM

• Fjalapeno moved this task from Backlog to Kanban on the Product-Infrastructure-Team-Backlog-Deprecated board.

• Fjalapeno edited projects, added Product-Infrastructure-Team-Backlog-Deprecated (Kanban); removed Product-Infrastructure-Team-Backlog-Deprecated.

• Fjalapeno claimed this task.Jul 19 2017, 4:27 PM

• Fjalapeno moved this task from To Do to Doing on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.

@bd808 I think Kiwix is using our Cloud VPS to generate Zim files. I assume that means we can do the same to generate some new files for our Android app?

Using the apps cloud VPS project in the near term

Actually we can't use Cloud VPS long term to do this if we want to upload to production Swift (also if we want to update the database of ZIm files in the MCS)

• Mholloway created subtask T172296: Determine how to gather top-viewed article lists for use in generating ZIM files.Aug 2 2017, 4:33 PM

• Mholloway closed subtask T172296: Determine how to gather top-viewed article lists for use in generating ZIM files as Resolved.Aug 3 2017, 12:03 AM

@Mholloway to provide stats on how long it takes to generate files and how long it takes to upload to Swift

• Fjalapeno renamed this task from Determine how to generate Zim Compilations using WMF infrastructure to Determine where to generate Zim Compilations using WMF infrastructure.Aug 7 2017, 4:43 PM

• Mholloway added a parent task: T172732: Create a production-ready zim content pack generation and upload service.Aug 7 2017, 7:41 PM

• Mholloway created subtask T172734: Collect data on zim file generation and upload times.Aug 7 2017, 7:44 PM

• Fjalapeno renamed this task from Determine where to generate Zim Compilations using WMF infrastructure to Determine where to generate Zim Compilations using Production WMF infrastructure.Aug 9 2017, 7:14 PM

• Fjalapeno updated the task description. (Show Details)

@ArielGlenn @bd808 after talking with @GWicke he suggested that the hardware used to generate dumps might fit the bill here. Do you think this is a possibility?

(Note: this is different to our previous conversation on where to host the files, that will be in Swift T170843: Determine where to host zim files for the Android app. This is only about generating the files)

It looks like this has been discussed before in other places:

https://www.mediawiki.org/wiki/Wikimedia_MediaWiki_Core_Team/Backlog/Improve_dumps#Architecture

T91853: Hardware for HTML / zim dumps

Here is the solution we are currently developing at openZIM/Kiwix. https://github.com/openzim/zimfarm. As far as I can see, this would fit to your needs. We plan to release a first beta in a week (fingers crossed).

• Mholloway renamed this task from Determine where to generate Zim Compilations using Production WMF infrastructure to Determine where to generate Zim content packs using Production WMF infrastructure.Aug 11 2017, 12:54 PM

@Kelson thanks that looks interesting, however this ticket is more about finding some production WMF hardware to run MWOffliner to generate ZIM files.

We may want to investigate this for use at some date in the future, but as you noted it is in beta currently, so we will probably wait a while and just use the tried and true MWOffliner in the mean time.

Thanks again

• Mholloway moved this task from Sign off to Blocked on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Nov 6 2017, 10:28 PM

• Mholloway edited projects, added Product-Infrastructure-Team-Backlog-Deprecated; removed Product-Infrastructure-Team-Backlog-Deprecated (Kanban).

• Mholloway moved this task from Needs triage to Needs investigation on the Product-Infrastructure-Team-Backlog-Deprecated board.

LGoto removed a project: Wikipedia-Android-App-Backlog.Apr 4 2018, 9:18 PM

• Mholloway moved this task from Needs investigation to Upcoming on the Product-Infrastructure-Team-Backlog-Deprecated board.Apr 6 2018, 7:09 PM

This is stalled, possibly indefinitely. Consider reopening if and when this work picks back up.

Dbrant closed subtask T172734: Collect data on zim file generation and upload times as Invalid.Aug 21 2018, 5:28 PM