Change Details

First of all, apologies from my side for some new work on the an-worker nodes :( In the parent task we realized that rack A4 has now 8 Hadoop worker nodes, I didn't realize that when planning the rack locations as described in T260445#6863444 I'd ask if you could help and try to balance some worker nodes again, since some of them are not yet in service and it would be easier for us to schedule downtime. If possible, I'd ask if any of these nodes could be racked in a different place: an-worker[1129,1139-1141].eqiad.wmnet The current distribution of hadoop worker nodes (without the aforementioned ones) is: ``` 19 A 19 B 21 C 19 D ``` Meanwhile the distribution between the rows is: ``` 1 A/1 5 A/2 2 A/3 4 A/4 2 A/5 5 A/7 5 B/2 1 B/3 5 B/4 5 B/7 3 B/8 5 C/2 4 C/3 7 C/4 4 C/7 1 C/8 6 D/2 4 D/4 2 D/5 6 D/7 1 D/8 ``` What we are trying to do is to avoid more than 5/6 nodes for each rack. Adding an-worker[1129,1139-1141] to A4 means getting to 8, that is too much for us (for resiliency if a rack goes down etc..). @wiki_willy proposed a change in T260445#6865096 to avoid IP changes (so keeping the nodes within row A), but if there are free spots elsewhere (in other rows) it will be fine as well. >>! In T260445#6865096, @wiki_willy wrote: > No worries @elukey, it looks like I missed the double count in rack A4 as well. If these hosts need to stay in row A though, the only other 10g options would be in racks A2 or A7. Both are pretty full, but I do see room to fit one server in each rack, near the very top. We typically don't use shelf 42, but it could be possible - @Jclark-ctr will probably need to confirm how tight the space is on shelf 42 is in A2 and A7. Also, ms-be1019 in A2 is EOL, so hopefully the SREs will have a decom task submitted for that soon, which would also free up another spot in the future. Would this work for you? > > - Move an-worker1129 to A2 > - Move an-worker1139 to A7 > > That would net you 6x servers in A2, 6x servers in A4, and 6x servers in A7. If it does, let's track this via a new task for the server moves. > > Thanks, > Willy > Thanks a lot for the patience!

First of all, apologies from my side for some new work on the an-worker nodes :( In the parent task we realized that rack A4 has now 8 Hadoop worker nodes, I didn't realize that when planning the rack locations as described in T260445#6863444 I'd ask if you could help and try to balance some worker nodes again, since some of them are not yet in service and it would be easier for us to schedule downtime. If possible, I'd ask if any of these nodes could be racked in a different place: an-worker[1129,1139-1141].eqiad.wmnet The current distribution of hadoop worker nodes (without the aforementioned ones) is: ``` 19 A 19 B 21 C 19 D ``` Meanwhile the distribution between the rows is: ``` 1 A/1 5 A/2 2 A/3 4 A/4 2 A/5 5 A/7 5 B/2 1 B/3 5 B/4 5 B/7 3 B/8 5 C/2 4 C/3 7 C/4 4 C/7 1 C/8 6 D/2 4 D/4 2 D/5 6 D/7 1 D/8 ``` What we are trying to do is to avoid more than 5/6 nodes for each rack. Adding an-worker[1129,1139-1141] to A4 means getting to 8, that is too much for us (for resiliency if a rack goes down etc..). @wiki_willy proposed a change in T260445#6865096 to avoid IP changes (so keeping the nodes within row A), but if there are free spots elsewhere (in other rows) it will be fine as well. >>! In T260445#6865096, @wiki_willy wrote: > No worries @elukey, it looks like I missed the double count in rack A4 as well. If these hosts need to stay in row A though, the only other 10g options would be in racks A2 or A7. Both are pretty full, but I do see room to fit one server in each rack, near the very top. We typically don't use shelf 42, but it could be possible - @Jclark-ctr will probably need to confirm how tight the space is on shelf 42 is in A2 and A7. Also, ms-be1019 in A2 is EOL, so hopefully the SREs will have a decom task submitted for that soon, which would also free up another spot in the future. Would this work for you? > > - Move an-worker1129 to A2 > - Move an-worker1139 to A7 > > That would net you 6x servers in A2, 6x servers in A4, and 6x servers in A7. If it does, let's track this via a new task for the server moves. > > Thanks, > Willy > Thanks a lot for the patience!

First of all, apologies from my side for some new work on the an-worker nodes :( In the parent task we realized that rack A4 has now 8 Hadoop worker nodes, I didn't realize that when planning the rack locations as described in T260445#6863444 I'd ask if you could help and try to balance some worker nodes again, since some of them are not yet in service and it would be easier for us to schedule downtime. If possible, I'd ask if any of these nodes could be racked in a different place: an-worker[1129,1139-1141].eqiad.wmnet The current distribution of hadoop worker nodes (without the aforementioned ones) is: ``` 19 A 19 B 21 C 19 D ``` Meanwhile the distribution between the rows is: ``` 1 A/1 5 A/2 2 A/3 4 A/4 2 A/5 5 A/7 5 B/2 1 B/3 5 B/4 5 B/7 3 B/8 5 C/2 4 C/3 7 C/4 4 C/7 1 C/8 6 D/2 4 D/4 2 D/5 6 D/7 1 D/8 ``` What we are trying to do is to avoid more than 5/6 nodes for each rack. Adding an-worker[1129,1139-1141] to A4 means getting to 8, that is too much for us (for resiliency if a rack goes down etc..). @wiki_willy proposed a change in T260445#6865096 to avoid IP changes (so keeping the nodes within row A), but if there are free spots elsewhere (in other rows) it will be fine as well. >>! In T260445#6865096, @wiki_willy wrote: > No worries @elukey, it looks like I missed the double count in rack A4 as well. If these hosts need to stay in row A though, the only other 10g options would be in racks A2 or A7. Both are pretty full, but I do see room to fit one server in each rack, near the very top. We typically don't use shelf 42, but it could be possible - @Jclark-ctr will probably need to confirm how tight the space is on shelf 42 is in A2 and A7. Also, ms-be1019 in A2 is EOL, so hopefully the SREs will have a decom task submitted for that soon, which would also free up another spot in the future. Would this work for you? > > - Move an-worker1129 to A2 > - Move an-worker1139 to A7 > > That would net you 6x servers in A2, 6x servers in A4, and 6x servers in A7. If it does, let's track this via a new task for the server moves. > > Thanks, > Willy > Thanks a lot for the patience!