Page MenuHomePhabricator

Load top article data into new AQS cluster
Closed, ResolvedPublic13 Estimated Story Points

Description

Rather than precomputing data: can we move the data directly from the old cluster?

Event Timeline

Nuria removed the point value for this task.

Can we use a python job to dump data and load into new cluster? Requires a bit of research.

We need to compare whether doing precomputations in hadoop is too costly so it is better to transfer data directly.

A possible compromise would be to load top data only for the last couple of months.

Nuria renamed this task from Load top data into new AQS cluster to Load top article data into new AQS cluster .Sep 8 2016, 4:01 PM
Nuria set the point value for this task to 13.Sep 8 2016, 4:04 PM

Full recomputation was actually fastest we could get.
I started full backfilling job last Friday that will finish either late tonight or early tomorrow.