Page MenuHomePhabricator
Create Task
Maniphest T202490

Automate XML-to-parquet transformation for XML dumps (oozie job)
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

This task involves:

Details

SubjectRepoBranchLines +/-
Add mediawiki-history-wikitext oozie jobanalytics/refinerymaster+493 -0
Add MediawikiXMLDumpsConverter spark jobanalytics/refinery/sourcemaster+433 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

JAllemandou created this task.Aug 22 2018, 7:54 AM
Milimetric triaged this task as High priority.Aug 23 2018, 4:04 PM
Milimetric moved this task from Incoming to Smart Tools for Better Data on the Analytics board.
JAllemandou added a project: Analytics-Kanban.Sep 4 2018, 4:14 PM
JAllemandou updated the task description. (Show Details)
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot subscribed.Sep 27 2018, 8:37 PM

Change 463370 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Add MediawikiXMLDumpsConverter spark job

https://gerrit.wikimedia.org/r/463370

gerritbot added a project: Patch-For-Review.Sep 27 2018, 8:37 PM
JAllemandou moved this task from In Progress to In Code Review on the Analytics-Kanban board.Sep 27 2018, 8:37 PM
MMiller_WMF unsubscribed.Sep 27 2018, 8:59 PM
gerritbot added a comment.Sep 28 2018, 8:18 PM

Change 463548 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Add mediawiki-history-wikitext oozie job

https://gerrit.wikimedia.org/r/463548

JAllemandou moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Oct 5 2018, 3:56 PM
gerritbot added a comment.Oct 8 2018, 12:07 PM

Change 463370 merged by jenkins-bot:
[analytics/refinery/source@master] Add MediawikiXMLDumpsConverter spark job

https://gerrit.wikimedia.org/r/463370

mforns set the point value for this task to 8.Oct 8 2018, 4:07 PM
gerritbot added a comment.Oct 22 2018, 2:14 PM

Change 463548 merged by Mforns:
[analytics/refinery@master] Add mediawiki-history-wikitext oozie job

https://gerrit.wikimedia.org/r/463548

• Nuria moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Oct 31 2018, 4:15 PM
• Nuria moved this task from Done to Ready to Deploy on the Analytics-Kanban board.
JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Nov 5 2018, 10:02 AM
• Nuria added a comment.Dec 10 2018, 5:58 PM

@JAllemandou I think we still need to document the table right? https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Mediawiki_wikitext_history

JAllemandou added a comment.Dec 10 2018, 7:23 PM

OMG ! Done ... Sorry for having skipped that one :S

• Nuria added a comment.Dec 10 2018, 9:12 PM

Ok, docs at : https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Mediawki_wikitext_history

• Nuria closed this task as Resolved.Dec 11 2018, 7:47 PM
diego added a comment.Feb 5 2019, 8:34 PM

Thanks @JAllemandou !

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits