Research spike: load enwiki data into Druid to study lookup table performance
Closed, DuplicatePublic21 Estimated Story Points
Actions

Assigned To

None

Authored By

	• Nuria
	Jul 27 2016, 7:15 PM

Description

Research spike: interested in lookup table performance and re-loading lookup tables with newly computed values. The challenge with the enwiki dataset is size, we estimate billions of values in the lookup tables.

Questions:

can we update lookup tables quickly enough (between the loading of each segment in Druid)
can we query Druid while joining to one or more lookup tables

If not, look into: https://clickhouse.yandex/

Desired outcome is some strategy to deal with these properties that change in the future. Valid strategy would be: don't do it and just compute it live every time you need it.

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T120037 Vital Signs: Please provide an "all languages" de-duplicated stream for the Community/Content groups of metrics
Resolved	None	T120036 Vital Signs: Please make the data for enwiki and other big wikis less sad, and not just be missing for most days
Resolved	odimitrijevic	T130256 Wikistats 2.0.
Resolved	None	T131779 Implement Pages Created & Count of Edits full vertical slice
Resolved	None	T131786 Load edit history data into Druid
Duplicate	None	T141472 Research spike: load enwiki data into Druid to study lookup table performance
Declined	JAllemandou	T134792 Spike - Slowly Changing Dimensions on Druid

Event Timeline

• Nuria created this task.Jul 27 2016, 7:15 PM

• Nuria added a subtask: T134792: Spike - Slowly Changing Dimensions on Druid.Jul 27 2016, 7:17 PM

Milimetric renamed this task from Research spike: load enwiki data into Druid to study whether we need lookup tables to Research spike: load enwiki data into Druid to study lookup table performance.Jul 28 2016, 5:28 PM

Milimetric updated the task description. (Show Details)

Milimetric set the point value for this task to 21.

Danny_B added a project: Spike.Jul 29 2016, 1:01 AM

Milimetric edited projects, added Analytics; removed Analytics-Kanban.Sep 15 2016, 3:44 PM

Milimetric moved this task from Incoming to Dashiki on the Analytics board.

Milimetric moved this task from Dashiki to Backlog (Later) on the Analytics board.Sep 15 2016, 4:40 PM

Isnt't this a duplicate of https://phabricator.wikimedia.org/T134792?

• Nuria closed this task as a duplicate of T134792: Spike - Slowly Changing Dimensions on Druid.Dec 12 2016, 5:14 PM

• Nuria closed subtask T134792: Spike - Slowly Changing Dimensions on Druid as Resolved.Mar 20 2017, 3:32 PM

Milimetric reopened subtask T134792: Spike - Slowly Changing Dimensions on Druid as Open.Mar 22 2017, 5:19 PM

• Nuria closed subtask T134792: Spike - Slowly Changing Dimensions on Druid as Declined.Mar 23 2017, 6:28 PM