Page MenuHomePhabricator

Configure Yarn to be able to locate nodes with a GPU
Open, Stalled, Needs TriagePublic

Description

In T255138 we added 6 nodes equipped with a GPU to the Hadoop cluster. As follow up, we should configure yarn to be able to locate and use those nodes when needed.

Yarn Labels may be a solution, but it should be reviewed/tested.

Event Timeline

elukey created this task.Oct 2 2020, 9:27 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2020, 9:27 AM

I found https://www.ibm.com/support/pages/node/6260093 that says:

Recommended versions
The YARN node labels feature was introduced in Apache Hadoop 2.6, but it’s not mature in the first official release. The recommended versions are 2.8 and later, which include a lot of fixes and improvements. For IOP, the supported version begins with IOP 4.2.5, which is based on Apache Hadoop 2.7.3. It has all the important fixes and improvements for node labels and has been thoroughly tested by us.

razzi changed the task status from Open to Stalled.Oct 15 2020, 4:07 PM
razzi added a subscriber: razzi.

@elukey says this depends on BigTop for the hadoop version.

razzi edited projects, added Analytics-Clusters; removed Analytics.Oct 15 2020, 4:07 PM