Page MenuHomePhabricator

Refactor webrequest refinement using Spark {hawk}
Closed, DeclinedPublic

Description

Webrequest refinement is currently done using Hive.
Refactoring it using Spark should bring performance improvement and allow wrapping multiple jobs together to take advantage of memory caching of data.

Event Timeline

JAllemandou claimed this task.
JAllemandou raised the priority of this task from to Normal.
JAllemandou updated the task description. (Show Details)
JAllemandou added subscribers: JAllemandou, Ottomata.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 1 2015, 6:48 PM

Let's task this before un-pausing at our tasking meeting.

kevinator closed this task as Declined.Jun 11 2015, 3:51 PM

At this moment, Hive is more stable than Spark for jobs that run regularly.
It would be nice to use Spark... but it's not mandatory and we don't need to take on additional risk. Declining task for now and it could be revisited in the future.

JAllemandou moved this task from Paused to Done on the Analytics-Kanban board.Feb 8 2016, 6:41 PM