@RobH @wiki_willy
Attached 3 photos Unfortunately large cooling fins have fitment issues in dell case. Any future gpu can not extend past backplane bracket

@Jclark-ctr Thanks for the photos. For clarification, does this mean the GPU does not fit or that the GPU does fit but any future GPU's order should be smaller in dimensions?

Gpu does not fit.

@Jclark-ctr Got it. Thanks.

@RobH @wiki_willy Can we set up a call to discuss next steps?

In T267050#6726182, @calbon wrote:

@Jclark-ctr Got it. Thanks.

@RobH @wiki_willy Can we set up a call to discuss next steps?

Sure. Additionally, my understanding of the GPU market is that any given GPU has multiple manufacters for the GPU. This was a "XFX - AMD Radeon RX 5700 XT 8GB GDDR6 256Bit PCIE, 246601", so we may be able to find another manufacter of this GPU chipset, which may have better/smaller form factor for within the case.

This is exactly why we only wanted to order 1 to test! =]

Additionally I've created T271351 to determine if there is another version of this GPU chipset via Rahi that may have a differing fan/heatsink assembly that would suit our needs. Discussion for that can take place on task T271351.

Pleae note that after our sync up with @wiki_willy and @calbon, the followup steps are:

• new GPU will be ordered for tested via T271351.
  • due to having to order another test GPU, the actuall rollout of GPUs through the ml-serve fleet will be delayed.
• systems will be racked and imaged without GPUs, as they can perform a sub-set of their intended function without the GPU. (It will just be less powerful, and not do quite as much.)
• cryptocurrency mining boom has put strain on worldwide gpu market, so sourcing a replacement gpu for a month or so may be problematic, so the system installs will proceed without GPUs for now
  • once the gpu arrives, whenever it is installed into a system, that system will need to be reimaged to ensure the GPU can be supported at time of system image/reimage in the future.

This had a mistake, introduced by me in the racking task, of 10G networking. These have 1G/10G nics, but ONLY need 1G. I had updated the codfw racking task, but failed to update this racking task.

So as a result, @Jclark-ctr had already racked 2 of them in 10G racks as of today, and will need to relocate them into 1G racks. Also updating that row diversity is prefered, but if one row in eqiad is overpopulated due to cloud, then two servers may have to share a row (but none should share a rack.)

Host racked ,working on cabling, netbox updated.
host port rack
ml-serve1001 12 a1
ml-serve1002 36 b5
ml-serve1003 5 c3
ml-serve1004 27 d8

Ran the dns cookbook for these hosts fine, but the homer script has issues:

robh@cumin1001:~$ sudo homer asw*eqiad* diff
INFO:homer.devices:Initialized 35 devices
INFO:homer:Generating diff for query asw*eqiad*
INFO:homer:Gathering global Netbox data
INFO:homer.devices:Matched 4 device(s) for query 'asw*eqiad*'
INFO:homer:Generating configuration for asw2-a-eqiad.mgmt.eqiad.wmnet
ERROR:homer.transports.junos:Failed to get diff for asw2-a-eqiad.mgmt.eqiad.wmnet: ConfigLoadError(severity: error, bad_element: 12, message: error: invalid interface name in 12
error: error recovery ignores input until this point
warning: mgd: statement must contain additional statements
error: invalid interface type: 12
error: error recovery ignores input until this point
warning: mgd: statement has no contents; ignored)
INFO:homer:Generating configuration for asw2-b-eqiad.mgmt.eqiad.wmnet
ERROR:homer.transports.junos:Failed to get diff for asw2-b-eqiad.mgmt.eqiad.wmnet: ConfigLoadError(severity: error, bad_element: 36, message: error: invalid interface name in 36
error: error recovery ignores input until this point
warning: mgd: statement must contain additional statements
error: invalid interface type: 36
error: error recovery ignores input until this point
warning: mgd: statement has no contents; ignored)
INFO:homer:Generating configuration for asw2-c-eqiad.mgmt.eqiad.wmnet
ERROR:homer.transports.junos:Failed to get diff for asw2-c-eqiad.mgmt.eqiad.wmnet: ConfigLoadError(severity: error, bad_element: 5, message: error: invalid interface name in 5
error: error recovery ignores input until this point
warning: mgd: statement must contain additional statements
error: invalid interface type: 5
error: error recovery ignores input until this point
warning: mgd: statement has no contents; ignored)
INFO:homer:Generating configuration for asw2-d-eqiad.mgmt.eqiad.wmnet
ERROR:homer.transports.junos:Failed to get diff for asw2-d-eqiad.mgmt.eqiad.wmnet: ConfigLoadError(severity: error, bad_element: 27, message: error: invalid interface name in 27
error: error recovery ignores input until this point
warning: mgd: statement must contain additional statements
error: invalid interface type: 27
error: error recovery ignores input until this point
warning: mgd: statement has no contents; ignored)
Changes for 4 devices: ['asw2-a-eqiad.mgmt.eqiad.wmnet', 'asw2-b-eqiad.mgmt.eqiad.wmnet', 'asw2-c-eqiad.mgmt.eqiad.wmnet', 'asw2-d-eqiad.mgmt.eqiad.wmnet']
# Failed
---------------
ERROR:homer:Homer run had issues on 4 devices: ['asw2-a-eqiad.mgmt.eqiad.wmnet', 'asw2-b-eqiad.mgmt.eqiad.wmnet', 'asw2-c-eqiad.mgmt.eqiad.wmnet', 'asw2-d-eqiad.mgmt.eqiad.wmnet']

host configured and racked

fixed the netbox issue, hsots will be imaged later today

In T267050#6732006, @RobH wrote:

fixed the netbox issue, hsots will be imaged later today

I hadn't circled back to this yet, but its on my radar to address/image today!

So I went to image these, and none of the mgmt interfaces are pingable. So either they aren't plugged in, or they were misconfigured. Since they aren't remotely accessible, I cannot troubleshoot this.

We'll need either @Jclark-ctr or @Cmjohnson to investigate and update the task description about the hosts being mgmt configured.

Change 655793 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] updating puppet repo for ml-serve100[1-4]

https://gerrit.wikimedia.org/r/655793

Change 655793 merged by RobH:
[operations/puppet@production] updating puppet repo for ml-serve100[1-4]

https://gerrit.wikimedia.org/r/655793

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

ml-serve1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202101131809_robh_27635_ml-serve1001_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['ml-serve1001.eqiad.wmnet']

Of which those FAILED:

['ml-serve1001.eqiad.wmnet']

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['ml-serve1001.eqiad.wmnet', 'ml-serve1002.eqiad.wmnet', 'ml-serve1003.eqiad.wmnet', 'ml-serve1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202101131946_robh_10085.log.

Completed auto-reimage of hosts:

['ml-serve1002.eqiad.wmnet', 'ml-serve1001.eqiad.wmnet', 'ml-serve1003.eqiad.wmnet', 'ml-serve1004.eqiad.wmnet']

Of which those FAILED:

['ml-serve1002.eqiad.wmnet', 'ml-serve1001.eqiad.wmnet', 'ml-serve1003.eqiad.wmnet', 'ml-serve1004.eqiad.wmnet']

So 100[123] failed the final part of the reimage script with puppet run, no clue why, need to investigate.

1004 is failing to get a DHCP response for PXE boot, and isn't being seen by the installer host request log/syslog. The switch shows the port online, so I need to investigate that as well.

Overall I've been fighting these all day to make them do what I want, so I'm quite frustrated and need to step away from them for a short bit.

Change 655997 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] correcting ml-serve1004 mac

https://gerrit.wikimedia.org/r/655997

Change 655997 merged by RobH:
[operations/puppet@production] correcting ml-serve1004 mac

https://gerrit.wikimedia.org/r/655997

ml-serve1--4 will pxe boot now.

next step is use image script again and investigate its errors.

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

ml-serve1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202101142000_robh_6361_ml-serve1001_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['ml-serve1001.eqiad.wmnet']

and were ALL successful.

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['ml-serve1002.eqiad.wmnet', 'ml-serve1003.eqiad.wmnet', 'ml-serve1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202101142059_robh_23189.log.

Completed auto-reimage of hosts:

['ml-serve1002.eqiad.wmnet', 'ml-serve1004.eqiad.wmnet', 'ml-serve1003.eqiad.wmnet']

and were ALL successful.