|Open||None||T73660 Add ZIM format support to OCG|
|Invalid||Kelson||T96235 Create a Debian package of zimwriterfs|
|Open||None||T69540 Produce/preserve the metadata about additional ResourceLoader modules required by extension tags|
|Resolved||marcoil||T73490 Parsoid should set the prop parameter when calling API action=expandtemplates|
|Resolved||marcoil||T86902 Improve Parsoid's loading of CSS modules using ResourceLoader|
|Invalid||Minervaxox||T115907 Outreachy proposal for T73660: Add ZIM format support to OCG|
|Open||None||T114788 OCG should download resourceLoader js/css dependencies|
|Declined||Adishaporwal||T116482 Outreachy proposal for Add ZIM format support to OCG|
- Mentioned In
- T172732: Create a production-ready zim content pack generation and upload service
T116482: Outreachy proposal for Add ZIM format support to OCG
T97672: Allow exporting MediaWiki pages in EPUB format
T94147: Bring back ZIM export in OCG toolchain
- Mentioned Here
- T150871: [EPIC] (Proposal) Replicate core OCG features and sunset OCG service
T161312: Decide what to do with open OCG tasks in Phabricator
rEOCB55aa6bea33e2: Save details of title redirects followed.
T69540: Produce/preserve the metadata about additional ResourceLoader modules required by extension tags
T96235: Create a Debian package of zimwriterfs
CScott - If I were a good programmer, I'd dive right in at the coding level to help because its a HUGE gap here, not having .zim output of WikiPedia, and not having easy Admin level .zim output from MediaWiki. I'm happy to do the legwork of trying to find you some qualified help. What programming language are your working in (or what are the choices) and do you need help understanding the .zim file format? I've toured http://www.openzim.org and http://www.kiwix.org trying to find active participants. Emmanuel Engelhart kelson at kiwix.org & Tommi Maekitalo tommi at tntnet.org might be able to help get the ball rolling.
Personally, I'm looking to pack Kiwix and .zim files as the primary off-line help system for OSMand (the extremely popular GPS / mobile mapping system that uses Open Street Maps off-line). The help files for all this will be authored on the Open Street Map MediaWiki and spun out to be the off-line help system on the Android, IOS and Destop (java) versions of the software. The global / any language capabilities are really attractive. All projects are free/open source. Kiwix is an obvious solution as long as zim files are easily produced on the fly by MediaWiki admins. Since maps are huge, and cell coverage spotty or completely non existent in many areas, off-line use of maps (and the help system) is critical. So our need dovtails nicely with the whole zim "philosophy".
Some scribbled notes below to perhaps uncover what the next best steps might be? I can't imagine the Kiwix people are happy about having zero zim output from WikiPedia and every other MediaWiki system.... PDF the only alternative? I cringe :)
This is the temporary solution suggested for the casual MediWiki admin to create .zim files?
- HTML export of your wiki pages to a directory that contains the entire assortment of related files:
- https://www.mediawiki.org/wiki/Extension:DumpHTML (however Release status: unstable)
- See also:
- Assemble the directory files into a zim file
- http://download.kiwix.org/dev/ZIMmakerVMv3.ova (that file is over 1gig, Open Virtualization Format (OVA) for software to be run in virtual machines. Will a run of the mill MediWiki/Linux admin be able to work with this?)
http://www.openzim.org/wiki/Build_your_ZIM_file has some tool suggestions:
- zim-postgresql.sql however doesn't seem to exist?
I also value being able to create zim files directly from mediawiki sites. I'm already spread too thin to be able to help practically unless my circumstances change materially. However I am involved in kiwix and willing to help test and debug whatever's involved. Emmanuel knows how to wake me up if/when the time is right :)
PS: I'm also helping create a wikibook for educational use (Computing aimed at 11 to 14 year old pupils in the UK schooling system) where it'd be great to make the book available as a zim file.
Here's what we have:
- Skeleton code to parse CLI options, unpack bundles, interface with OCG in https://github.com/wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-zim_renderer
- It is missing the code which actually transforms the HTML for local viewing, but this part can be borrowed from http://sourceforge.net/p/kiwix/other/ci/master/tree/mwoffliner/
- After it writes the standalone HTML tree on disk, you would invoke zimwriterfs to actually create the ZIM file (but see T96235: Create a Debian package of zimwriterfs).
IIRC, last time I looked at the code, some tweaks to https://github.com/wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-bundler might be needed as well. I already added one in 55aa6bea33e29053b76b2043d2c96bcb2f4f1964 since the zimwriter backend needs to rewrite redirects. I believe there were other minor issues involving stylesheets & etc -- for example, the Parsoid DOM includes a stylesheet URL, but we don't actually fetch it in the bundler. (And in this case a better solution would be to use the API to query the actual style modules necessary, instead of just stashing the result of ResourceLoader; see T69540: Produce/preserve the metadata about additional ResourceLoader modules required by extension tags). I'm happy to do the mw-ocg-bundler side of this work; just create phab tickets for specific items and link them here as blockers.
I am shifting this to Outreachy-Round-11 as the project description has two mentors, micro-tasks and looks ready for the 11th edition of Outreachy ( Dec 2015 - Mar 2016 ) . Potential candidates should start by submitting their proposals as a blocker for this task, by November 02.
Feel free to revert it back, if this task has some relevant issues which might block its completion in this term of Outreachy.
I would like to work on this project as a part of Outreachy round 12/ GSoC 2016. I am fairly good at PHP and know some Node.JS. I have read about the ZIM format and OCG. I intend to look at the present stub of solution implemented in the next couple days, and in parallel solve the microtasks. @cscott, will you be willing to work as a mentor for this project?
I am a software engineering student and i am quite new to WIkimedia.
While browsing the possible projects, i read through this project and It seems very interesting. I am willing to take this project during this GsoC '17 Please @cscott if you agree with that I can move ahead directly with looking deeply at the project. Thanks
@Aklapper @Eugene233 On our side this is still pretty important even if we have no focus on this due to lack of resources. I have posted a comment in that direction here https://phabricator.wikimedia.org/T146757#2959943. That said, to the contrary to the OCG, the electron-renderer (effort) seems be self-focused an to offer little opportunities to be reused for other formats.
@cscott Thx for confirming my feeling.
"mwoffliner" is not available as a npm module, so it can be directly/easily used in OCG.
We are currently fixing the problem with mocking the resourceLoader for offline usage in mwoffliner and use also the mobile layout. This should be finished in a few weeks.
Then, it would be smart to move away from zimwriterfs binary call, and use directly node-libzim. One time that's made, it should be relatively easy to bring ZIM export in OCG.
"New Reader" and "global reach" teams are pretty supportive to that feature AFAIK and this is important to Kiwix project too. Looks like we just need to gather supportive people to get enough support to get dev resources to "finish the job".