@Aroraakhil there are two ways of checking metrics:

1. sudo radeontop
2. https://grafana.wikimedia.org/d/ZAX3zaIWz/amd-rocm-gpu?orgId=1

rocm-smi is unfortunately a python script that requires full root to execute, so we decided to use radeontop.

@elukey thanks much for your response. However, none of these monitoring tools give information about the pids of the processes or the number of processes currently using the GPU. 'nvidia-smi' provides that, and thus, I am assuming its equivalent 'rocm-smi' should also provide that.
Is there no reasonable workaround to install 'rocm-smi'?

Best,
Akhil

@Aroraakhil this is the output of rocm-smi (I executed manually via sudo) by default is:

elukey@stat1005:~$ sudo /opt/rocm/bin/rocm-smi


 ========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK     MCLK    Fan    Perf  PwrCap  VRAM%  GPU%
1    48.0c  87.0W   1500Mhz  945Mhz  14.9%  auto  170.0W  N/A    100%
================================================================================
==============================End of ROCm SMI Log ==============================

Anything in particular that you were used with nvidia-smi that I should check? (namely, can you give me an example?)

We can think about allowing rocm-smi, but it is a script (not a binary) and we'd need to give it sudo/root permissions, something not really great security wise. Let's figure out if it is really needed, if so I'll try do to something :)

@elukey thanks much for your prompt response. This is what I get from 'nvidia-smi' in our EPFL machine. As you can see it displays the number of processes currently running, and the pids. However, I am not sure if there is a flag in rocm-smi that displays similar information.

I see /opt/rocm-3.3.0/bin/rocm-smi --showpids that could help, will investigate!

Thanks much!

@elukey just curious if there are any updates on this?

@Aroraakhil sorry still no progress, I hope to get something done this Quarter :(

@elukey thanks!

I did some testing just now, and it looks like the current version of rocm_smi.py does not try to re-execute itself through sudo when the --showpidgpus or --showpids flags are used. Luca tells me that this used to be the case, but it looks like it has changed since. @Aroraakhil can you test whether the tool now works for you?

@elukey and @klausman thanks!
It works fine for me.
Just to clarify, I use the following /opt/rocm/bin/rocm-smi --showpids without sudo, and it works just fine. I am assuming this is how it is supposed to be, rite?