Page MenuHomePhabricator
Create Task
Maniphest T247082

Upgrade AMD ROCm to latest upstream
Closed, ResolvedPublic
Actions

Description

We are currently running ROCm version 2.7.1, and the last upstream version is 3.10 (https://rocm-documentation.readthedocs.io/en/latest/Current_Release_Notes/Current-Release-Notes.html).

While testing with @Miriam we found some bugs of the combination tensorflow-rocm / ROCm that we hope to see it fixed after the upgrade (worst case scenario we'll provide a bug report to upstream with their last version of the code).

Before starting, it is worth to point out that the new version will probably only support tensorflow-rocm 2.x. Waiting for @Miriam's green light before proceeding :)

Details

SubjectRepoBranchLines +/-
profile::statistics::gpu: upgrade stat1005 to rocm 3.3operations/puppetproduction+2 -8
Add rocm-smi path as parameter to the Prometheus AMD GPU exporteroperations/puppetproduction+13 -5
Fix AMD GPU prometheus exporter for ROCm 3.3operations/puppetproduction+2 -1
amd: update package list after 3.3 upgradeoperations/puppetproduction+1 -2
profile::statistics::gpu: upgrade stat1008 to ROCm 3.3operations/puppetproduction+10 -3
aptrepo: add configuration for AMD ROCm 3.3operations/puppetproduction+11 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Mar 6 2020, 2:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2020, 2:09 PM
Milimetric triaged this task as High priority.Mar 9 2020, 4:01 PM
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.
elukey mentioned this in T248574: GPUs are not correctly handling multitasking .Mar 26 2020, 1:09 PM
gerritbot added a comment.Apr 6 2020, 8:35 AM

Change 586310 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] aptrepo: add configuration for AMD ROCm 3.3

https://gerrit.wikimedia.org/r/586310

gerritbot added a project: Patch-For-Review.Apr 6 2020, 8:35 AM
gerritbot added a comment.Apr 6 2020, 10:56 AM

Change 586310 merged by Elukey:
[operations/puppet@production] aptrepo: add configuration for AMD ROCm 3.3

https://gerrit.wikimedia.org/r/586310

Maintenance_bot removed a project: Patch-For-Review.Apr 6 2020, 11:10 AM
Stashbot added a comment.Apr 6 2020, 11:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-06T11:18:28Z] <elukey> import AMD ROCm 3.3 packages in buster-wikimedia (component thirdparty/rocm33) - T247082

gerritbot added a comment.Apr 6 2020, 11:18 AM

Change 586334 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::statistics::gpu: upgrade stat1008 to ROCm 3.3

https://gerrit.wikimedia.org/r/586334

gerritbot added a project: Patch-For-Review.Apr 6 2020, 11:18 AM
gerritbot added a comment.Apr 6 2020, 11:39 AM

Change 586334 merged by Elukey:
[operations/puppet@production] profile::statistics::gpu: upgrade stat1008 to ROCm 3.3

https://gerrit.wikimedia.org/r/586334

Maintenance_bot removed a project: Patch-For-Review.Apr 6 2020, 12:10 PM
gerritbot added a comment.Apr 6 2020, 1:19 PM

Change 586352 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] amd: update package list after 3.3 upgrade

https://gerrit.wikimedia.org/r/586352

gerritbot added a project: Patch-For-Review.Apr 6 2020, 1:20 PM
gerritbot added a comment.Apr 6 2020, 1:22 PM

Change 586352 merged by Elukey:
[operations/puppet@production] amd: update package list after 3.3 upgrade

https://gerrit.wikimedia.org/r/586352

Maintenance_bot removed a project: Patch-For-Review.Apr 6 2020, 2:10 PM
gerritbot added a comment.Apr 6 2020, 2:18 PM

Change 586362 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Fix AMD GPU prometheus exporter for ROCm 3.3

https://gerrit.wikimedia.org/r/586362

gerritbot added a project: Patch-For-Review.Apr 6 2020, 2:18 PM
gerritbot added a comment.Apr 6 2020, 2:23 PM

Change 586362 merged by Elukey:
[operations/puppet@production] Fix AMD GPU prometheus exporter for ROCm 3.3

https://gerrit.wikimedia.org/r/586362

elukey added a project: Analytics-Kanban.Apr 6 2020, 2:36 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
Maintenance_bot removed a project: Patch-For-Review.Apr 6 2020, 3:10 PM
elukey added a comment.Apr 9 2020, 12:50 PM

From Miriam's tests it seems that stat1008 (with ROCm 3.3) works better than before, with some positive effects also on T248574. Since some tests are ongoing on stat1005 I'll update it in a couple of weeks.

elukey moved this task from In Progress to Paused on the Analytics-Kanban board.Apr 9 2020, 12:50 PM
gerritbot added a comment.May 14 2020, 2:40 PM

Change 596457 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add rocm-smi path as parameter to the Prometheus AMD GPU exporter

https://gerrit.wikimedia.org/r/596457

gerritbot added a project: Patch-For-Review.May 14 2020, 2:40 PM

Change 596457 merged by Elukey:
[operations/puppet@production] Add rocm-smi path as parameter to the Prometheus AMD GPU exporter

https://gerrit.wikimedia.org/r/596457

Maintenance_bot removed a project: Patch-For-Review.May 14 2020, 3:10 PM
Milimetric moved this task from Paused to In Progress on the Analytics-Kanban board.Jun 8 2020, 4:42 PM
MGerlach subscribed.Jun 8 2020, 4:49 PM
gerritbot added a comment.Jun 9 2020, 7:10 AM

Change 603814 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::statistics::gpu: upgrade stat1005 to rocm 3.3

https://gerrit.wikimedia.org/r/603814

gerritbot added a project: Patch-For-Review.Jun 9 2020, 7:10 AM
gerritbot added a comment.Jun 9 2020, 7:12 AM

Change 603814 merged by Elukey:
[operations/puppet@production] profile::statistics::gpu: upgrade stat1005 to rocm 3.3

https://gerrit.wikimedia.org/r/603814

elukey added a comment.Jun 9 2020, 7:32 AM

stat1005 upgraded! Docs updated as well https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/AMD_GPU

elukey set Final Story Points to 13.Jun 9 2020, 7:41 AM
elukey moved this task from In Progress to Done on the Analytics-Kanban board.
Maintenance_bot removed a project: Patch-For-Review.Jun 9 2020, 8:10 AM
elukey added a project: Analytics-Clusters.Jun 10 2020, 2:41 PM
elukey moved this task from Backlog to Q4 2019/2020 on the Analytics-Clusters board.Jun 10 2020, 2:43 PM
• Nuria subscribed.Jun 19 2020, 4:57 PM

making sure @Miriam is not waiting for any work here before proceeding

elukey added a comment.Jun 19 2020, 4:58 PM

Didn't add a message in here, but the upgrade is completed, I synced with Miriam and Martin (who uses the GPU) before proceeding, all good.

• Nuria closed this task as Resolved.Jun 19 2020, 5:03 PM
Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM
klausman closed subtask T248574: GPUs are not correctly handling multitasking as Resolved.Sep 17 2020, 1:27 PM
klausman reopened subtask T248574: GPUs are not correctly handling multitasking as Open.Sep 17 2020, 1:29 PM
klausman closed subtask T248574: GPUs are not correctly handling multitasking as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL