Page MenuHomePhabricator

rack new hadoop worker nodes
Closed, ResolvedPublic

Description

Parent Task T100442 was for the request of new hadoop worker nodes. As they have arrived, this task will track their rack placement, racking, & setup.

Event Timeline

RobH claimed this task.
RobH raised the priority of this task from to Medium.
RobH updated the task description. (Show Details)
RobH edited projects, added ops-eqiad; removed hardware-requests.
RobH set Security to None.
RobH added subscribers: Matanya, Cmjohnson, Dzahn and 5 others.

Last instruction from @Ottomata is to place 4 in rack d2-eqiad for now, the location of the remainder still need to be determined.

3 of 4 are racked in row D and connected to mgmt. They only need
re-install. The 4th one was broken and I didn't get to replace it before I
left on vacation. Dell has sent me a replacement and I will get that up
next week for you.

As for the other 12, I don't really know where we're going to put them yet.
I know there are a bunch of db's I am going to decom in the next couple of
weeks in row A and B. That will open up space for a few but I don't have
the details yet.

Hm, oh! Chris, we can move analytics1003, analytics1004 and analytics1010 to Row D. If I remember correctly those are tall servers, so maybe we can fit a few in their spots?

Those are the 3 remaining Ciscos that we still use. We don't use them for production stuff, but we do use them for testing eventlogging changes and for stream processing evaluations, so I'd like to keep them as long as they aren't broken. But, they can be powered down and moved at any time.

analytics1042-1045 are racked and ready for install in row D2. Racktables has been updated.

analytics1045.mgmt.eqiad.wmnet has address 10.65.4.17
analytics1044.mgmt.eqiad.wmnet has address 10.65.4.16
analytics1043.mgmt.eqiad.wmnet has address 10.65.4.15
analytics1042.mgmt.eqiad.wmnet has address 10.65.4.14

analytics1042.eqiad.wmnet has address 10.64.53.22
analytics1043.eqiad.wmnet has address 10.64.53.23
analytics1044.eqiad.wmnet has address 10.64.53.24
analytics1045.eqiad.wmnet has address 10.64.53.25

Change 226102 had a related patch set uploaded (by Ottomata):
Add host entries for analytics1042-analytics1045

https://gerrit.wikimedia.org/r/226102

Change 226102 merged by Ottomata:
Add host entries for analytics1042-analytics1045

https://gerrit.wikimedia.org/r/226102

1042-1045 have base install w/out puppet certs.

1042-1045 are installed and part of the Hadoop cluster.

1046-1049 are racked and setup in row B and ready for installs

DNS/DHCP/Raid Cfg/Switch Cfg has been completed

Change 228343 had a related patch set uploaded (by Ottomata):
Provision analytics1046-1049 as Hadoop worker nodes

https://gerrit.wikimedia.org/r/228343

Change 228343 merged by Ottomata:
Provision analytics1046-1049 as Hadoop worker nodes

https://gerrit.wikimedia.org/r/228343

All are racked and @Ottomata has already added as worker nodes. Unfortunately, we had 2 more servers that were came to us not working. Dell tech support is replacing them and we should see them next week. I will update with expected arrival times.

Until then I am going to leave this open. Also Racktables is pending completion.

Received 1 of 2 replacement servers

All have been racked and setup...the replacements 1053 and 1057 have not been installed yet. I will leave that up to @Ottomata

I just installed and puppetized these nodes. Thanks!