In response to New Readers [[ https://meta.wikimedia.org/wiki/New_Readers/Findings | research ]] showing the need for better offline support, as well as a well-received Community Wishlist [[ https://meta.wikimedia.org/wiki/2016_Community_Wishlist_Survey/Categories/Reading#CW2016-R077 | proposal ]], the Wikipedia Android app team is working on improving the offline user experience by adding support for loading Wikipedia articles from [[ http://www.openzim.org/wiki/ZIM_file_format | ZIM ]] files. On the app side, the technical work is nearly complete; searching and loading articles from one or more ZIM files loaded onto the device works well, and only productization work (i.e., user onboarding and integration with the rest of the app's functionality) remains to be done.
In addition to developing our general knowledge and competency around this technology, there are a couple of areas for exploration and possible improvement around the **content** of the ZIM files currently created by the [[ https://github.com/openzim/mwoffliner | mwoffliner ]] tool, which is used to create most if not all of the ZIM files currently available:
- We'd like to expand beyond Kiwix’s library of ZIM files. (T169905)
- The HTML content of the articles in the existing ZIM files has a lot of Kiwix-specific formatting, which the Wikipedia app needs to strip before displaying it.[1] Ideally the articles in the ZIM file shouldn't be adulterated in any way, and should be identical to the content received if it a network request were made to get the same article. (T172764)
- We'd like to expand upon and and improve the metadata that is "baked into" the various ZIM files offered to our users. This metadata is what the user sees when deciding which compilation to download, so it must be worded very clearly and meaningfully. (T164760)
There are also concerns around **hosting** ZIM files to be downloaded in-app:
- We need to use infrastructure that we can scale for hosting content that we serve to our apps. That means that we need to find production WMF hardware to host the ZIM files that we serve. After some internal discussion, [[ https://docs.openstack.org/swift/latest/ | Swift ]], the service used to host all of the media content uploaded to [[ https://commons.wikimedia.org | Wikimedia Commons ]], has emerged as a strong candidate for hosting these files, and we need to test Swift’s capacity for handling and serving files of this size. (T172123)
- Wikimedia's production Swift service does not permit uploading files from [[ https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS | Cloud VPS ]], and therefore if Swift is indeed used for hosting ZIM files, they'll have to be uploaded from a ZIM file generation service set up consistently with the requirements for running in the Wikimedia production environment. Therefore we're working on prototyping an mwoffliner instance set up as it would be if running in Wikimedia production.
To the extent we desire changes in mwoffliner our goal is to work closely with OpenZIM/Kiwix to develop them in a generally useful way and contribute our work back upstream.
[1] E.g., the unexpected license footer here: {F9034591}