Page MenuHomePhabricator

New Hadoop hardware. Refreshes and hosts with space for GPUs
Closed, DuplicatePublic

Description

New Hadoop hardware. Refreshes and hosts with space for GPUs.

  • refreshes for 1042-1057. This budget comes from budgeted refreshes
  • new nodes with space for 6 GPUs. This budget comes from underrun

Sounds like hadoop nodes use 10G links and space is scarce on datacenter so we need to plan how to do this.

Event Timeline

@Nuria I added some comments to T238587#5800760, please let me know if it makes sense or not.

Just to confirm the amount of 10g vs 1g NICs:

elukey@cumin1001:~$ sudo cumin 'A:hadoop-worker' 'cat /sys/class/net/*/speed 2>&1 | grep -v "Invalid"'
54 hosts will be targeted:
an-worker[1078-1095].eqiad.wmnet,analytics[1042-1077].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(34) analytics[1042-1075].eqiad.wmnet
----- OUTPUT of 'cat /sys/class/n...rep -v "Invalid"' -----
1000
===== NODE GROUP =====
(20) an-worker[1078-1095].eqiad.wmnet,analytics[1076-1077].eqiad.wmnet
----- OUTPUT of 'cat /sys/class/n...rep -v "Invalid"' -----
10000

Some things to check:

  1. analytics1058->68 should be refreshed as well this fiscal
  2. what about analytics1070->75? They seem to be an expansion done later, that should not be refreshed before 2022. This means that eventually these hosts will work with 1G in a cluster of all 10Gs.

In any case, it seems that we'd need to move a lot of hosts to 10G, and we might not have space in the racks for this move.