Page MenuHomePhabricator
Create Task
Maniphest T120983

Develop data loading scripts for the Research Cluster (altiscale)
Closed, ResolvedPublic
Actions

Description

This task is done when we have a single-command ETL for converting a new XML dump into a query-able set of HIVE tables.

On the altiscale "Research" cluster.

See current work here: https://github.com/wikimedia-research/research-cluster

See old notes here: https://etherpad.wikimedia.org/p/research_cluster_loading

Full process: [XML Dump] --> [JSON files] --> [Hive Table] --> [Metadata Hive Table]

[XML Dump] --> [JSON files] is handled by dump2revdocs.py

[Hive Table] & [Metadata Hive Table] is handled by some HiveQL scripts

Related Objects
Search...

Event Timeline

Halfak created this task.Dec 9 2015, 6:50 PM
Halfak claimed this task.
Halfak raised the priority of this task from to Needs Triage.
Halfak updated the task description. (Show Details)
Halfak added a project: Research.
Halfak moved this task to In Progress on the Research board.
Halfak added subscribers: Halfak, JAllemandou.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 9 2015, 6:50 PM
Halfak added a comment.Jan 14 2016, 11:07 PM

I finally got python installed on the workbench! (Every time I tried over the last month, the workbench's drive was full and I couldn't do anything)

Next steps are tests with real data.

Halfak added a parent task: T123688: Onboarding Nathaniel: software engineering tasks.Jan 21 2016, 11:24 PM
Halfak updated the task description. (Show Details)Jan 21 2016, 11:33 PM
Halfak set Security to None.
Halfak reassigned this task from Halfak to • schana.Jan 26 2016, 3:09 PM
Halfak updated the task description. (Show Details)
Halfak updated the task description. (Show Details)
Halfak added a subscriber: • schana.
Halfak updated the task description. (Show Details)Jan 26 2016, 3:12 PM
Halfak added a comment.Feb 9 2016, 9:54 PM

I just gave a quick intro to @schana and we worked for a little while on getting the script up and running. See our work here: https://github.com/wikimedia-research/research-cluster

JAllemandou added a comment.Feb 10 2016, 3:23 PM

First version of a dump_download script --> https://github.com/wikimedia-research/research-cluster/pull/1

• DarTar added a project: Measuring-value-added.Feb 18 2016, 8:45 PM
Kenrick95 moved this task from In Progress to Done (current quarter) on the Research board.Mar 1 2016, 12:45 AM
Kenrick95 moved this task from Done (current quarter) to In Progress on the Research board.Mar 1 2016, 12:21 PM
• DarTar triaged this task as Medium priority.Mar 17 2016, 10:29 PM
Halfak added a comment.Apr 28 2016, 10:27 PM

@JAllemandou, it seems like this is done now. What do you think?

• ggellerman closed this task as Resolved.Apr 28 2016, 10:27 PM
• ggellerman reassigned this task from • schana to JAllemandou.
• ggellerman moved this task from In Progress to Done (current quarter) on the Research board.
JAllemandou added a comment.Apr 29 2016, 8:34 AM

Agreed @Halfak , first version is done and merged !

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL