Page MenuHomePhabricator
Create Task
Maniphest T287267

Update ROCm version on GPU instances.
Closed, ResolvedPublic
Actions

Description

While evaluating pre-built ML models for a portion of T250436 one of the most promissing we evaluated is provided through onnxruntime. This runs acceptably on CPU's, but ideally it would run on our GPU instance as well. ROCm support in onnxruntime is provided through MIGraphX, AMD's inference acceleration engine. Reviewing the ROCm repositories, MIGraphX was added to the ROCm repositories in 3.9, but we are currently on 3.8. It's not clear exactly what version of MIGraphX is required, but updating to at least ROCm 3.9 would allow evaluating.

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

EBernhardson created this task.Jul 23 2021, 4:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 23 2021, 4:55 PM
Ottomata edited projects, added Analytics; removed Analytics-Clusters.Jul 26 2021, 3:34 PM
fkaelin subscribed.Aug 4 2021, 4:29 PM
elukey added a project: Machine-Learning-Team.Aug 4 2021, 4:32 PM
• ACraze subscribed.Aug 4 2021, 4:36 PM
• ACraze added a comment.Aug 4 2021, 4:44 PM

+1 for upgrading ROCm to support ONNX runtime. It's certainly worth evaluating imo, as it seems that ONNX would help enable us to use an AMD GPU with any arbitrary ML-framework

kevinbazira subscribed.Aug 4 2021, 4:52 PM
elukey moved this task from Unsorted to Backlog/SRE on the Machine-Learning-Team board.Aug 6 2021, 6:28 AM
odimitrijevic removed a project: Machine-Learning-Team.EditedAug 23 2021, 4:36 PM
odimitrijevic subscribed.

@elukey is this work that the ML team plans on implementing/owning?

odimitrijevic moved this task from Incoming to Analytics Query Service on the Analytics board.Aug 23 2021, 6:08 PM
odimitrijevic moved this task from Analytics Query Service to Incoming on the Analytics board.
elukey added a comment.Aug 24 2021, 7:08 AM

@odimitrijevic yes definitely we can work on it (either by ourselves or working with Ben/Razzi if they are not super busy with other projects). Lemme know :)

odimitrijevic moved this task from Incoming to Machine Learning Platform on the Analytics board.Sep 2 2021, 5:06 PM
odimitrijevic edited projects, added Analytics-Radar; removed Analytics.
odimitrijevic added subscribers: BTullis, • razzi.

That's great! @BTullis and @razzi are busy but do reach out if you have any questions.

gerritbot added a comment.Oct 4 2021, 1:31 PM

Change 725887 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] amd_rocm: import ROCm suite 4.3.1

https://gerrit.wikimedia.org/r/725887

gerritbot added a project: Patch-For-Review.Oct 4 2021, 1:32 PM
elukey added a comment.Oct 4 2021, 1:36 PM

Things to review: 3.8 -> 4.3.1

https://rocmdocs.amd.com/en/latest/Current_Release_Notes/ROCm-Version-History.html

https://rocmdocs.amd.com/en/latest/Current_Release_Notes/Current-Release-Notes.html#amd-rocm-release-notes-v4-3
https://rocmdocs.amd.com/en/latest/Current_Release_Notes/ROCm-Version-History.html#new-features-and-enhancements-in-rocm-v4-2
https://rocmdocs.amd.com/en/latest/Current_Release_Notes/ROCm-Version-History.html#new-features-and-enhancements-in-rocm-v4-1
https://rocmdocs.amd.com/en/latest/Current_Release_Notes/ROCm-Version-History.html#new-features-and-enhancements-in-rocm-v4-0
https://rocmdocs.amd.com/en/latest/Current_Release_Notes/ROCm-Version-History.html#new-features-and-enhancements-in-rocm-v3-10
https://rocmdocs.amd.com/en/latest/Current_Release_Notes/ROCm-Version-History.html#new-features-and-enhancements-in-rocm-v3-9

gerritbot added a comment.Oct 4 2021, 2:02 PM

Change 725887 merged by Elukey:

[operations/puppet@production] amd_rocm: import ROCm suite 4.3.1

https://gerrit.wikimedia.org/r/725887

gerritbot added a comment.Oct 4 2021, 2:09 PM

Change 725904 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] aptrepo: add missing amd-rocm431 settings

https://gerrit.wikimedia.org/r/725904

gerritbot added a comment.Oct 4 2021, 2:10 PM

Change 725904 merged by Elukey:

[operations/puppet@production] aptrepo: add missing amd-rocm431 settings

https://gerrit.wikimedia.org/r/725904

Stashbot added a comment.Oct 4 2021, 2:19 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-04T14:19:57Z] <elukey> import AMD ROCm 4.3.1 packages in buster-wikimedia's thirdparty/amd-rocm431 - T287267

Maintenance_bot removed a project: Patch-For-Review.Oct 4 2021, 3:10 PM
AikoChou subscribed.Oct 4 2021, 3:40 PM
gerritbot added a comment.Oct 5 2021, 6:25 AM

Change 726389 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Set new AMD ROCm version for an-worker1096

https://gerrit.wikimedia.org/r/726389

gerritbot added a project: Patch-For-Review.Oct 5 2021, 6:25 AM
gerritbot added a comment.Oct 5 2021, 7:01 AM

Change 726507 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] amd_rocm: update settings/packages for ROCm 4.3.1

https://gerrit.wikimedia.org/r/726507

gerritbot added a comment.Oct 5 2021, 7:02 AM

Change 726507 merged by Elukey:

[operations/puppet@production] amd_rocm: update settings/packages for ROCm 4.3.1

https://gerrit.wikimedia.org/r/726507

gerritbot added a comment.Oct 5 2021, 7:03 AM

Change 726389 merged by Elukey:

[operations/puppet@production] Set new AMD ROCm version for an-worker1096

https://gerrit.wikimedia.org/r/726389

gerritbot added a comment.Oct 5 2021, 7:38 AM

Change 726539 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Upgrade all an-workers with GPUs to ROCm 4.3.1

https://gerrit.wikimedia.org/r/726539

gerritbot added a comment.Oct 5 2021, 7:40 AM

Change 726539 merged by Elukey:

[operations/puppet@production] Upgrade all an-workers with GPUs to ROCm 4.3.1

https://gerrit.wikimedia.org/r/726539

gerritbot added a comment.Oct 5 2021, 11:06 AM

Change 726578 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move stat100[5,8] to AMD ROCm 4.3.1

https://gerrit.wikimedia.org/r/726578

gerritbot added a comment.Oct 5 2021, 12:30 PM

Change 726606 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] amd_rocm: add support for ROCm 4.2

https://gerrit.wikimedia.org/r/726606

gerritbot added a comment.Oct 5 2021, 12:35 PM

Change 726606 merged by Elukey:

[operations/puppet@production] amd_rocm: add support for ROCm 4.2

https://gerrit.wikimedia.org/r/726606

gerritbot added a comment.Oct 5 2021, 12:42 PM

Change 726611 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Downgrade AMD ROCm to 4.2 (from 4.3.1) on an-worker1096

https://gerrit.wikimedia.org/r/726611

Stashbot added a comment.Oct 5 2021, 12:43 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-05T12:43:25Z] <elukey> import AMD ROCm 4.2 to buster-wikimedia's thirdparty/amd-rocm42 - T287267

gerritbot added a comment.Oct 5 2021, 12:44 PM

Change 726611 merged by Elukey:

[operations/puppet@production] Downgrade AMD ROCm to 4.2 (from 4.3.1) on an-worker1096

https://gerrit.wikimedia.org/r/726611

gerritbot added a comment.Oct 5 2021, 12:55 PM

Change 726619 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Downgrade AMD ROCm to 4.2 on all GPU-based Hadoop workers

https://gerrit.wikimedia.org/r/726619

gerritbot added a comment.Oct 5 2021, 1:00 PM

Change 726619 merged by Elukey:

[operations/puppet@production] Downgrade AMD ROCm to 4.2 on all GPU-based Hadoop workers

https://gerrit.wikimedia.org/r/726619

gerritbot added a comment.Oct 6 2021, 7:24 AM

Change 726759 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] prometheus-amd-rocm-stats.py: support ROCm 4.2.0's smi output

https://gerrit.wikimedia.org/r/726759

gerritbot added a comment.Oct 6 2021, 7:26 AM

Change 726759 merged by Elukey:

[operations/puppet@production] prometheus-amd-rocm-stats.py: support ROCm 4.2.0's smi output

https://gerrit.wikimedia.org/r/726759

elukey added a comment.Oct 6 2021, 7:52 AM

To keep archives happy:

  • We decided to target ROCm 4.3.1 (current latest upstream) and tensorflow-rocm 2.6.
  • Instead of rolling out the packages on stat100[5,8], we started from the Hadoop workers.
  • Basic checks after install went fine, but then we realized that tensorflow-rocm 2.6 (following its tensorflow counterpart) doesn't support anymore reading from HDFS natively. It needs a new package, tensorflow-io, that in turn requires tensorflow (non-rocm version). We'll need to follow up with upstream (AMD) to ask what are their plans.
  • We decided to target ROCm 4.2 and tensorflow 2.5.0
elukey added a comment.Oct 6 2021, 8:04 AM

https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/issues/1461

gerritbot added a comment.Oct 6 2021, 10:15 AM

Change 726864 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Upgrade stat100[5,8] to ROCm 4.2

https://gerrit.wikimedia.org/r/726864

gerritbot added a comment.Oct 6 2021, 10:16 AM

Change 726864 merged by Elukey:

[operations/puppet@production] Upgrade stat100[5,8] to ROCm 4.2

https://gerrit.wikimedia.org/r/726864

• ACraze mentioned this in T292306: [DSE Hackathon] Sounds of the Commons: Neural Audio Mashups.Oct 8 2021, 4:00 AM
gerritbot added a comment.Oct 12 2021, 6:13 AM

Change 726578 abandoned by Elukey:

[operations/puppet@production] Move stat100[5,8] to AMD ROCm 4.3.1

Reason:

https://gerrit.wikimedia.org/r/726578

elukey closed this task as Resolved.Oct 12 2021, 12:56 PM
elukey claimed this task.

This is done! It seems that ROCm 4.2 is the only viable option for the moment, I'll keep working on https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/issues/1461 to see what upstream suggests.

Please re-open the task if anything is missing!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits