Page MenuHomePhabricator

Identify (and potentially) help move mwoffliner off NFS
Closed, ResolvedPublic

Description

It's one of the biggest users of NFS at the moment. Figure out what exactly NFS is being used for, and find alternative solutions (and help implement it!)

Event Timeline

yuvipanda claimed this task.
yuvipanda raised the priority of this task from to Needs Triage.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Cloud-Services.
Kelson added a subscriber: Kelson.Jun 16 2015, 7:29 PM

http://sourceforge.net/p/kiwix/other/ci/master/tree/mwoffliner/ is what the project does - nodejs based tool. Only external dependencies are access to parsoid + MW API.

So the primary purpose of the project is to produce ZIM files of all the projects. NFS is used primarily to cache image files, and the ZIM files are then uploaded to the Kiwix mirror site (via rsync?).

I think the main requirement for the ZIM dump host is sufficient storage, and ideally some amount of isolation from production.

We currently have francium allocated for HTML dumps, but don't have full access to it as it is a production host without a way to really isolate it. This means that it will be hard to use it for ZIM generation unless we go through a lengthy security review.

It would be great to have support for large storage or bare metal instances in the labs subnet. ZIM dump generation in particular would become significantly simpler if we had access to a host like francium in labs.

They have moved to using /data/scratch now

yuvipanda closed this task as Resolved.Jun 26 2015, 3:43 PM

And all other NFS mounts have been disabled.