Make history and current wikitext available in hadoop
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	JAllemandou
	Nov 21 2019, 7:31 PM

Description

Add oozie job for pages_meta_current (dump imported but not converted to avro - copy of the job from pages_meta_history)
Bump docs (add pages_meta_current and siteinfo datasets, and add examples on how to use)

Details

	Subject	Repo	Branch	Lines +/-
	Update wikitext oozie job adding current	analytics/refinery	master	+147 -98
	Enforce distinct revision in xml-dumps converter	analytics/refinery/source	master	+3 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Milimetric	T186559 Provide data dumps in the Analytics Data Lake
		Resolved		JAllemandou	T238858 Make history and current wikitext available in hadoop

Event Timeline

JAllemandou created this task.Nov 21 2019, 7:31 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 21 2019, 7:31 PM

JAllemandou mentioned this in T234333: Import siteinfo dumps onto HDFS.Nov 22 2019, 9:49 AM

mforns triaged this task as Medium priority.Nov 25 2019, 5:12 PM

mforns moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

JAllemandou added a project: Analytics-Kanban.Jan 17 2020, 12:55 PM

JAllemandou updated the task description. (Show Details)

Change 565558 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Update wikitext oozie job adding current

https://gerrit.wikimedia.org/r/565558

Change 565554 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Enforce distinct revision in xml-dumps converter

https://gerrit.wikimedia.org/r/565554

Change 565554 merged by jenkins-bot:
[analytics/refinery/source@master] Enforce distinct revision in xml-dumps converter

https://gerrit.wikimedia.org/r/565554

JAllemandou moved this task from Next Up to Ready to Deploy on the Analytics-Kanban board.Feb 5 2020, 5:17 PM

JAllemandou claimed this task.Feb 5 2020, 5:23 PM

JAllemandou updated the task description. (Show Details)

JAllemandou set the point value for this task to 5.

Change 565558 merged by Joal:
[analytics/refinery@master] Update wikitext oozie job adding current

https://gerrit.wikimedia.org/r/565558

JAllemandou renamed this task from Update wikitext-processing on hadoop various aspects to Make history and current wikitext available in hadoop.Feb 5 2020, 8:34 PM

Maintenance_bot removed a project: Patch-For-Review.Feb 5 2020, 9:11 PM

JAllemandou added a parent task: T186559: Provide data dumps in the Analytics Data Lake.Feb 5 2020, 9:53 PM

JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Feb 5 2020, 9:58 PM

Do we need to update docs? https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Content/XMLDumps/Mediawiki_wikitext_history

It's been done a few days ago :)
https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Content/XMLDumps/Mediawiki_wikitext_current exists as well.

• Nuria closed this task as Resolved.Feb 25 2020, 9:04 PM

MGerlach mentioned this in T305688: Make HTML Dumps available in hadoop.Apr 8 2022, 12:39 PM