Maniphest T134791

Scale MySQL edit history reconstruction data extraction
Closed, ResolvedPublic0 Estimated Story Points
Actions

Assigned To

Authored By

	JAllemandou
	May 9 2016, 5:29 PM

Tags

Referenced Files

None

Subscribers

Description

Scale MySQL edit history reconstruction data extraction for page history /user history and revision history.

We will be using the same algorithm we have used to reconstruct history in simplewiki in enwiki, due to size we need an strategy to paralleize it.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T120037 Vital Signs: Please provide an "all languages" de-duplicated stream for the Community/Content groups of metrics
		Resolved		None	T120036 Vital Signs: Please make the data for enwiki and other big wikis less sad, and not just be missing for most days
		Resolved		• odimitrijevic	T130256 Wikistats 2.0.
		Resolved		• Nuria	T134791 Scale MySQL edit history reconstruction data extraction
		Resolved		mforns	T141468 User history: rewrite the user history script to use the new algorithm
		Resolved		JAllemandou	T141548 Scale scala algorithms using graph partitioning
		Resolved		JAllemandou	T141778 Use scalable algo on enwiki
		Resolved		JAllemandou	T142552 Productionize scala code for edit reconstruction
		Resolved		mforns	T142724 Page History: Add unit tests to PageHistoryDataExtractors and PageHistoryBuilder
		Resolved		JAllemandou	T144717 Extract edit history denormalized data from intermediate data

Event Timeline

JAllemandou created this task.May 9 2016, 5:29 PM

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 9 2016, 5:29 PM

JAllemandou added a project: Analytics.May 9 2016, 5:31 PM

JAllemandou updated the task description. (Show Details)

JAllemandou added a subtask: T134790: Extract edit oriented data from MySQL for simplewiki.

JAllemandou moved this task from Incoming to Operational Excellence Future on the Analytics board.May 9 2016, 5:37 PM

• Nuria moved this task from Operational Excellence Future to Backlog (Later) on the Analytics board.May 16 2016, 4:55 PM

• Nuria moved this task from Backlog (Later) to Operational Excellence Future on the Analytics board.

nshahquinn-wmf subscribed.May 19 2016, 9:39 PM

Akeron subscribed.May 27 2016, 3:32 PM

Milimetric added a parent task: T130256: Wikistats 2.0..Jul 21 2016, 4:37 PM

• Nuria set the point value for this task to 0.Jul 21 2016, 5:06 PM

• Nuria edited projects, added Analytics-Kanban; removed Analytics.

• Nuria moved this task from Next Up to Parent Tasks on the Analytics-Kanban board.

• Nuria renamed this task from Scale MySQL edit data extraction to Scale MySQL edit history reconstruction data extraction .Jul 27 2016, 7:19 PM

• Nuria updated the task description. (Show Details)

• Nuria created subtask T141476: Productionize edit history extraction for all wikis using Sqoop.Jul 27 2016, 7:29 PM

• Nuria assigned this task to JAllemandou.Jul 28 2016, 4:13 PM

JAllemandou removed a subtask: T134790: Extract edit oriented data from MySQL for simplewiki.Jul 28 2016, 5:40 PM

JAllemandou added a subtask: T141468: User history: rewrite the user history script to use the new algorithm.

JAllemandou created subtask T141548: Scale scala algorithms using graph partitioning.

JAllemandou created subtask T141778: Use scalable algo on enwiki.Aug 1 2016, 3:54 PM

• Nuria closed subtask T141468: User history: rewrite the user history script to use the new algorithm as Resolved.Aug 4 2016, 4:08 PM

• Nuria closed subtask T141778: Use scalable algo on enwiki as Resolved.

JAllemandou created subtask T142552: Productionize scala code for edit reconstruction.Aug 10 2016, 6:46 AM

• Nuria closed subtask T141548: Scale scala algorithms using graph partitioning as Resolved.Aug 15 2016, 3:11 PM

JAllemandou removed JAllemandou as the assignee of this task.Sep 5 2016, 8:31 AM

JAllemandou created subtask T144717: Extract edit history denormalized data from intermediate data.

• Nuria closed subtask T142552: Productionize scala code for edit reconstruction as Resolved.Oct 3 2016, 3:04 PM

outdated, no longer relevant

• Nuria removed a subtask: T141476: Productionize edit history extraction for all wikis using Sqoop.Nov 30 2016, 8:03 PM

• Nuria closed subtask T144717: Extract edit history denormalized data from intermediate data as Resolved.Mar 22 2017, 7:50 PM