Page MenuHomePhabricator

Test sqooping from the new dedicated labsdb host
Closed, ResolvedPublic5 Story Points

Description

We should first make sure sqooping works using that box as it does on the other labsdb-analytics ones.
Then we should test some performance improvements using more mappers for bi projects, and sqooping more small projects at a time.

The new labsdb host, labsdb1012.eqiad.wmnet, is ready for testing. The only tunable that we have is the limit of concurrent TCP connections, set to 10 in the "regular" labsdb environment. I have already set it to '100' for the Analytics user, let's see if it is enough or if we need more (verifying that the host can handle the load of course).

Details

Related Gerrit Patches:
analytics/refinery : masterUpdate refinery sqoop to use dedicated labsdb host
operations/puppet : productionUpdate sqoop timers configuration
analytics/refinery : masterUpdate refinery sqoop parallel execution

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2019, 8:52 PM
Nuria triaged this task as High priority.Feb 11 2019, 4:34 PM
Nuria moved this task from Incoming to Operational Excellence on the Analytics board.
elukey updated the task description. (Show Details)
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 495266 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Update refinery sqoop to use dedicated labsdb host

https://gerrit.wikimedia.org/r/495266

Change 496560 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Update refinery sqoop parallel execution

https://gerrit.wikimedia.org/r/496560

Change 495266 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Update refinery sqoop to use dedicated labsdb host

https://gerrit.wikimedia.org/r/495266

Change 499409 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Update sqoop timers configuration

https://gerrit.wikimedia.org/r/499409

Change 499409 merged by Elukey:
[operations/puppet@production] Update sqoop timers configuration

https://gerrit.wikimedia.org/r/499409

Change 495266 merged by Milimetric:
[analytics/refinery@master] Update refinery sqoop to use dedicated labsdb host

https://gerrit.wikimedia.org/r/495266

Nuria closed this task as Resolved.Apr 4 2019, 8:15 PM
Nuria set the point value for this task to 5.