Bump @RobH

So there are not going to be a lot of server chassis that can accomodate an off the shelf GPU card. Additionally, it would then not have any kind of warranty support for the card included in the system chassis.

I'd recommend we buy the GPU included with the system, which limits our availability in GPU models. We presently purchase Dell and HP systems.

The Dell offerings only have the Dell PowerEdge R730 as a system we would typically want to run in operations. We already use the R730xd system for other analytics servers, this is just a modified (less disk bays being the most major modification) of that chassis. The reduction in disk bays allows for a longer expansion card bays, so they can accomodate these cards:

http://www.dell.com/learn/us/en/04/campaigns/poweredge-gpu

All of the available Dell GPU options are there. I'd like to get some feedback from @Halfak & @DarTar on if any of those offerings are preferred (I'd assume nvida).

I've sent an email off to Dasher to have them provide me a detailed breakdown of the GPU offerings in the DL system line.

Edit Addition: I should include some background info for why I've put in the above criteria:

• We want to stick with Dell or HP, since those are the system vendors we presently deploy in our datacenters.
• we want to stick with the same server line within a vendor if possible. So that means the PowerEdge R line with Dell, and the DL line with HP. Shifting lines often causes some growing pains (Dell C2100 fiasco), so its best to avoid when possible.

Once we get the GPU options hammered down, all the rest of the specs are easy in comparison.

Please also note a concern was raised about the driver support of these GPU options:

In T148843#3075519, @Ladsgroup wrote:

Regarding GPU options. I just want to note that their drivers are propriety software and not open source (or partially open source with very limited options for HPC) except Intel® Xeon Phi™ coprocessor 3120P but number of cores in that GPU is smaller than other ones. Wondering if we can buy two and use both in one machine for extreme computing power.

This is a relevant concern.

Note that the Nvidia OpenCL drivers are closed-source (as the other parts of the Nvidia drivers). Note sure about AMD, but they've become far more FOSS-friendly in recent years.

What's the status of this (and stat1003 too)? Who is this waiting on? Is it @Ottomata or @RobH?

stat1003 ticket is T159839. This stat1002 replacement ticket is waiting on feedback from @DarTar and @Halfak about acceptable GPU specs.

OK I did some digging. My assessment is that the Nvidia Tesla K80 is most desirable, but the closed source drivers will be a problem. The AMD FirePro S9150 is a possibly good alterative, but support for AMD gpus is "experimental" and I can't find any clear evidence that it actually *works* with tensor flow.

Notes:

I see the Nvidia Tesla K80 listed at http://www.dell.com/learn/us/en/04/campaigns/poweredge-gpu This is one of the recommended GPUs for Tensor Flow -- one of the ML technologies that we are targeting.

We're going to have issues with Nvidia's drivers because they are closed source. Not sure if I could say if that's a blocker or just something we're going to feel bad about. It seems that if Nvidia's drivers were not a problem, we'd definitely want to go Nvidia.

Nvidia seems to be the top recommendation.

• http://timdettmers.com/2017/03/19/which-gpu-for-deep-learning/ "The thing is that the GPU computing or GPGPU community is very large for CUDA and rather small for OpenCL."
• http://stackoverflow.com/questions/40000518/is-intel-based-graphic-card-compatible-with-tensorflow-gpu "No"

Still, there's an experimental port of tensorflow for supporting OpenCL and therefor AMD drivers

• https://github.com/benoitsteiner/tensorflow-opencl Purports to allow basic support in an opencl environment.
  • "TensorFlow can only take advantage of accelerators that support OpenCL 1.2. Supported accelerators include but are not limited to: AMD Fiji, AMD Hawaii"
• Note that AMD FirePro S9150 supports "OpenCL 1.2 conformance expected" and uses the "Hawaii" chip architecture.

I'd say the closed source driver is a blocker. We've blocked the use of some PCIe flash memory cards due to closed source driver usage in years past, so I'd assume its still a blocker.

I'd ask @MoritzMuehlenhoff or @faidon to comment perhaps? I'd work with the assumption that closed source is not allowed.

I'll plan on getting some quotes generated with the Dell and HP options, with the more open source friendly GPU options to start. If the closed source is approved, then I'll get the quotes updated/appended to include those as needed.

Leaving our FLOSS policy aside, the proprietary Nvidia is going to be a significant problem both in terms of

• maintainability (we tend to run more recent kernels on top of our base distributions, and even while the nvidia drivers are provided in Debian non-free, they will be packaged for the kernel driver in Debian stable). Plus Debian stretch will provide signed kernel images by default and I have no idea whether that's compatible with external drivers modules.
• security (it's a multi megabyte blog of closed source kernel code, over which we don't have any control)

How about Beignet, the open source drivers for Intel cards, would that work for your use case?
https://www.freedesktop.org/wiki/Software/Beignet/
https://packages.qa.debian.org/b/beignet.html

Yeah, we had a similar conversation over email with Adam (@dr0ptp4kt) who was also inquiring about TensorFlow. I had the same considerations that were mentioned above about NVidia binary drivers and CUDA: basically that it's both against our FL/OSS policy and also has issues with maintenance/tainting the kernel and security.

Since there is an alternative (the AMD GPU), and it doesn't sound like it would impossible to use or a ton of extra effort, and it would save us probably quite a bit of extra effort and risk on the infrastructure side, I think we a) owe it to ourselves to stick to our principles and try using hardware that has free libre drivers and b) we owe it to the FL/OSS community at large to report back and support the TensorFlow OpenCL efforts (if not us, then who?). If it turns out that it's impossible to use AMD for this, we can always step back, reconsider, and replace it with the NVidia card. It's a single GPU AIUI and the server could remain the same, so the extra cost wouldn't be impossible to bear and at that point, we would know that we gave the better option our best.

@Halfak/@dr0ptp4kt, what do you think? Do you feel comfortable with the plan above and with commiting with potentially the extra effort of using the AMD GPU?

I've emailed with a contact re: OpenCL support for the Nvidia Tesla P100 (presumably facilitated by mainstreaming of OpenCL), and also shared this ticket with the mention of AMD FirePro S9150 with OpenCL (or absent it if that's a possibility without encumbered drivers).

To be clear, it's likely that using AMD/opencl could involve "a ton of extra effort", but I don't see a good alternative given what @MoritzMuehlenhoff and @faidon have said. And hey, maybe we'll get lucky! I think we should go that direction. It seems that support for opencl in tensor flow is growing fast if it's not already going to be nice and easy for us.

@faidon, Aaron pointed that I should have a look at this conversation given that some of the work we may do with TensorFlows in the coming year may be affected by the decision on this ticket. Your proposal makes sense to me and I'm happy to give it a try.

Thanks all! :)

@RobH, see the section under Disks in the task description. The storage requirements aren't so strict, but as usual, more is better. stat1002 is using ~5T of space. We should give ourselves a little room to grow. I'd say, 8T of available space would be plenty.

@Ottomata @RobH this seems to have been stalled somewhere between you two. Could you guys figure this and and T159839 out this week? Thanks!

In T159838#3171078, @faidon wrote:

@Ottomata @RobH this seems to have been stalled somewhere between you two. Could you guys figure this and and T159839 out this week? Thanks!

It stalled out with me, I'll get the quote requests off for this right now.

We've already gotten quotes back for T159839 and I've just escalated it up for reviews.

@Halfak, @leila. We've got some quotes back from vendors, and have a choice between the AMD FirePro S9150 and the AMD FirePro S7150. The way the numbers and specs line up, I'm inclined to go with a quote that includes the S9150. I just want to check with yall to be sure that given these 2 options, you don't actually prefer the S7150 over the S9150. Does the S9150 sound good to you?

@Ottomata Bob and I reviewed and we are happy with your choice.

Great, thanks!

+1