Page MenuHomePhabricator

Test sqooping from the new dedicated labsdb host
Closed, ResolvedPublic5 Story Points

Description

We should first make sure sqooping works using that box as it does on the other labsdb-analytics ones.
Then we should test some performance improvements using more mappers for bi projects, and sqooping more small projects at a time.

The new labsdb host, labsdb1012.eqiad.wmnet, is ready for testing. The only tunable that we have is the limit of concurrent TCP connections, set to 10 in the "regular" labsdb environment. I have already set it to '100' for the Analytics user, let's see if it is enough or if we need more (verifying that the host can handle the load of course).

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2019, 8:52 PM
Nuria triaged this task as High priority.
elukey updated the task description. (Show Details)
JAllemandou claimed this task.

Change 495266 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Update refinery sqoop to use dedicated labsdb host

https://gerrit.wikimedia.org/r/495266

Change 496560 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Update refinery sqoop parallel execution

https://gerrit.wikimedia.org/r/496560

Change 495266 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Update refinery sqoop to use dedicated labsdb host

https://gerrit.wikimedia.org/r/495266

Change 499409 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Update sqoop timers configuration

https://gerrit.wikimedia.org/r/499409

Change 499409 merged by Elukey:
[operations/puppet@production] Update sqoop timers configuration

https://gerrit.wikimedia.org/r/499409

Change 495266 merged by Milimetric:
[analytics/refinery@master] Update refinery sqoop to use dedicated labsdb host

https://gerrit.wikimedia.org/r/495266

Nuria set the point value for this task to 5.Apr 4 2019, 8:15 PM
Nuria closed this task as Resolved.