umbrella task for all platform design related questions that need to be investigated and answered for the wikiproject (and other) dumps 2.0 rewrite
see https://www.mediawiki.org/wiki/Talk:Wikimedia_Developer_Summit_2016/T114019/Minutes/Questions for source
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | None | T146070 Review progress of dumps rewrite | |||
| Open | None | T128513 Dumps 2.0 Platform design questions | |||
| Open | None | T128520 What already available software can we build on for a job scheduler? | |||
| Open | None | T143205 Draft usage scenarios for job/workflow manager | |||
| Open | None | T143206 List requirements needed for task/job/workflow manager. | |||
| Open | None | T143207 Evaluate software packages for job/task/workflow management | |||
| Open | None | T128755 What monitoring of dumps can we provide to users? | |||
| Open | None | T128754 Can we use a stream of events for facilitating incrementals of the project content dumps? | |||
| Open | None | T128752 What sort of internal storage do we want for the new dumps sytem? |
@ArielGlen: I think before getting too deep in the dumps rewrite you should take a look at the work done by analytics to reconstruct the edit history from mediwiki, you could benefit from all that work. Maybe you can get in touch with @Ottomata at the ops meeting? On my opinion it will be a waste of resources to rebuild a workflow system when we already have one deployed working at scale that already knows how to read the mediawiki database.
cc @mark