Page MenuHomePhabricator

Load top article data into new AQS cluster
Closed, ResolvedPublic13 Story Points

Description

Rather than precomputing data: can we move the data directly from the old cluster?

Event Timeline

Nuria created this task.Sep 8 2016, 3:57 PM
Nuria removed the point value for this task.
Nuria added a comment.EditedSep 8 2016, 4:00 PM

Can we use a python job to dump data and load into new cluster? Requires a bit of research.

We need to compare whether doing precomputations in hadoop is too costly so it is better to transfer data directly.

A possible compromise would be to load top data only for the last couple of months.

Nuria renamed this task from Load top data into new AQS cluster to Load top article data into new AQS cluster .Sep 8 2016, 4:01 PM
Nuria set the point value for this task to 13.Sep 8 2016, 4:04 PM

Full recomputation was actually fastest we could get.
I started full backfilling job last Friday that will finish either late tonight or early tomorrow.

JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.
Nuria closed this task as Resolved.Sep 19 2016, 3:11 PM