Page MenuHomePhabricator

Pull data for edit reconstruction from labs
Closed, ResolvedPublic

Description

Can we use just 1 wiki? (to set aside scooping/availability issues?)

The idea of analytics scooping data from the new Labsdb cluster was discussed on mw summit and it has the approval of chase and moritz. Let's use this as the parent task to organize work.

UPDATE: Half complete as of May 2017, still need to push data back to labs for access via Quarry, etc.

Event Timeline

Nuria created this task.Dec 9 2016, 5:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 9 2016, 5:58 PM
Nuria edited projects, added Analytics; removed Analytics-Kanban.Dec 9 2016, 5:59 PM
Nuria moved this task from Incoming to Dashiki on the Analytics board.Dec 12 2016, 4:53 PM
Nuria updated the task description. (Show Details)Jan 17 2017, 4:38 PM
Nuria added subscribers: Joalpe, chasemp, Milimetric.
Nuria edited subscribers, added: JAllemandou; removed: Joalpe.

I believe we discussed (for now) extracting data from the (new) labsdb hosts (1009-1011) instead of sanitarium (which would be a different prospect)

Nuria updated the task description. (Show Details)Jan 17 2017, 5:10 PM
Nuria renamed this task from POC of clickhouse + denormalize edit data on labs that can be used as a backend for quarry to Pull data for edit reconstruction from labs. .Jan 17 2017, 6:16 PM

Correct @chasemp !
It would be great if we could do a test run while the servers are not yet busy with real labs users, it would allow to measure the impact of sqooping out data.
Here the task for the first one-off: T155658.
I plan to add subtasks as we move forward.

JAllemandou renamed this task from Pull data for edit reconstruction from labs. to Pull data for edit reconstruction from labs and push it back after reconstruction.Jan 19 2017, 1:17 PM
Milimetric triaged this task as Medium priority.
Milimetric updated the task description. (Show Details)
Nuria added a comment.Jul 3 2017, 4:52 PM

I think this task can be closed now right @JAllemandou ?

@Nuria : The pull rom labs is done, the push is not.
So if we want to close, let's scope-down this one and create av new one about pushing to labs.

Nuria renamed this task from Pull data for edit reconstruction from labs and push it back after reconstruction to Pull data for edit reconstruction from labs .Jul 3 2017, 6:55 PM
Nuria closed this task as Resolved.