Build out hadoop job to calculate average page views over time for cirrussearch scoring purposes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	EBernhardson
	Oct 20 2015, 5:57 PM

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Initial popularity score calculator	wikimedia/discovery/analytics	master	+623 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	EBernhardson	T116055 Build out hadoop job to calculate average page views over time for cirrussearch scoring purposes
Resolved	• dcausse	T113437 Make the CirrusSearch index updaters take advantage of "adding fields for free"
Resolved	• dcausse	T116016 Adjust rescoring methods to be able to optionally use additional fields for scoring information
Resolved	Smalyshev	T113440 build out infra to be able to take data calculated in hadoop and import it to the elasticsearch cluster on a weekly basis
Resolved	EBernhardson	T116023 Add page_id to pageview_hourly when present in webrequest x_analytics header
Resolved	akosiaris	T120281 Make elasticsearch cluster accessible from analytics hadoop workers

Event Timeline

EBernhardson created this task.Oct 20 2015, 5:57 PM

EBernhardson raised the priority of this task from to Needs Triage.

EBernhardson updated the task description. (Show Details)

EBernhardson added a project: CirrusSearch.

EBernhardson subscribed.

Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptOct 20 2015, 5:57 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

EBernhardson added subtasks: T113437: Make the CirrusSearch index updaters take advantage of "adding fields for free", T116016: Adjust rescoring methods to be able to optionally use additional fields for scoring information, T113440: build out infra to be able to take data calculated in hadoop and import it to the elasticsearch cluster on a weekly basis, T116023: Add page_id to pageview_hourly when present in webrequest x_analytics header.Oct 20 2015, 5:58 PM

• Deskana closed subtask T116016: Adjust rescoring methods to be able to optionally use additional fields for scoring information as Resolved.Nov 24 2015, 5:49 PM

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 24 2015, 5:49 PM

• Deskana closed subtask T113437: Make the CirrusSearch index updaters take advantage of "adding fields for free" as Resolved.Nov 24 2015, 5:49 PM

• Deskana added a project: Discovery-Search (Current work).Nov 24 2015, 5:59 PM

• Deskana set Security to None.

EBernhardson claimed this task.Nov 26 2015, 1:03 AM

EBernhardson moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.

• Deskana triaged this task as Medium priority.Nov 26 2015, 1:47 AM

• Deskana subscribed.

I started building this by putting together a hive query, but after playing with it a bit I don't think it's going to be flexible enough. Oozie can kick of spark jobs, so working that out now. Spark can be done with the java, scala or python bindings. Not sure which to use, but the docs seem to lean towards scala so will work with that initially.

actually since stas started his part in python, it seems to make the most sense to continue and do all our spark work in python unless a specific need arises to diverge.

Change 256167 had a related patch set uploaded (by EBernhardson):
Initial popularity score calculator

https://gerrit.wikimedia.org/r/256167

gerritbot added a project: Patch-For-Review.Dec 1 2015, 5:38 PM

• Nuria closed subtask T116023: Add page_id to pageview_hourly when present in webrequest x_analytics header as Resolved.Dec 2 2015, 6:46 PM

Smalyshev mentioned this in T120281: Make elasticsearch cluster accessible from analytics hadoop workers.Dec 3 2015, 8:59 PM

Smalyshev added a subtask: T120281: Make elasticsearch cluster accessible from analytics hadoop workers.

Smalyshev mentioned this in T120729: Create monitoring for Spark jobs.Dec 7 2015, 10:32 PM

EBernhardson moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Dec 8 2015, 5:39 PM

Change 256167 merged by Smalyshev:
Initial popularity score calculator

https://gerrit.wikimedia.org/r/256167

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Dec 8 2015, 11:34 PM

• Deskana closed this task as Resolved.Dec 17 2015, 11:51 PM