Page MenuHomePhabricator

Troubleshoot data warehouse importing process {mole}
Closed, ResolvedPublic

Description

The process of importing data into the warehouse is not working as expected.

Instead of vetting the data and passing the problems to Sean Pringle,
we can troubleshoot the problems ourselves using a provisional warehouse instance
by just importing a small chunk of data (couple of days/weeks) from production.

This way, we can identify errors in the importing scripts quicker,
and finally, when we find the scripts are correct, pass them to Sean
to execute them full scope and import on the real warehouse.

Event Timeline

mforns created this task.Feb 4 2015, 6:33 PM
mforns claimed this task.
mforns raised the priority of this task from to High.
mforns updated the task description. (Show Details)
mforns added a project: Analytics-Kanban.
mforns added a subscriber: mforns.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2015, 6:33 PM
mforns lowered the priority of this task from High to Normal.Feb 4 2015, 6:33 PM
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.
mforns set Security to None.
kevinator renamed this task from Troubleshoot data warehouse importing process to Troubleshoot data warehouse importing process {vole}.Feb 5 2015, 1:26 AM
kevinator renamed this task from Troubleshoot data warehouse importing process {vole} to Troubleshoot data warehouse importing process {mole}.Feb 5 2015, 1:35 AM
gerritbot added a subscriber: gerritbot.

Change 189532 had a related patch set uploaded (by Mforns):
Adapt loading and automatic verification scripts

https://gerrit.wikimedia.org/r/189532

Patch-For-Review

kevinator closed this task as Resolved.Apr 27 2015, 4:27 PM
kevinator added a subscriber: kevinator.

Investigation is done.
now looking into some inserts have data from the past T76075: LabsDB problems negatively affect analytics tools like Wikimetrics, Vital Signs, Quarry, etc. {mole}

kevinator moved this task from Paused to Done on the Analytics-Kanban board.Apr 27 2015, 11:59 PM

Change 189532 abandoned by Mforns:
Adapt loading and automatic verification scripts

Reason:
This project is not in the road map any more.

https://gerrit.wikimedia.org/r/189532