ml-serve1013 and ml-serve1014 are now DPX/NPS1, 16x96GB each, and both advertise amd.com/gpu: 16 to Kubernetes. ml-serve1012 (SPX/NPS1, 8x192GB) and ml-serve1015 (CPX/NPS4, 64x24GB) keep their layouts.

Why the first attempt was reverted

amd-k8s-device-plugin 1.31.0.8 advertised one device per physical card in every partition mode. It matched each partition to its parent card through the KFD unique_id property, assuming the two share a value. On these hosts every partition has a distinct unique_id (64 of them for 64 partitions), so no partition ever matched, and every secondary partition was dropped.

Upstream fixed this in 1.31.0.9 by grouping partitions on location_id/domain instead. I imported 1.31.0.10 into operations/debs/amd-k8s-device-plugin and published 1.31.0.10-1 to trixie-wikimedia. I stopped short of 1.31.0.11 because it needs Go 1.26.7, which only trixie-backports has.

Side effect worth noting: until the upgrade, ml-serve1013-1015 exposed 8x24GB, i.e. 192GB of each node's 1536GB, to Kubernetes. ml-serve1015 went from 8 to 64 allocatable on the upgrade alone.

MI300X partitioning, as observed on this hardware

• The supported modes are SPX/NPS1, DPX/NPS1, QPX/NPS1-4 and CPX/NPS1-4. NPS2 is not available, although AMD's docs list DPX+NPS2, so 96GB partitions mean DPX+NPS1.
• Memory-mode writes are only staged until amd-smi reset -r (a driver reload, 1-2 minutes).
• A driver reload does not keep the compute mode. It falls back to the default for the memory mode (SPX for NPS1, CPX for NPS4), so the compute mode has to be set after the reload. Setting it live needs no reload.
• After a reboot, the memory mode is kept and the compute mode is not.
• amd-smi exits 0 even when it rejects a write, so the script now reads state back from sysfs instead of trusting the exit code.
• amd-smi GPU indices count partitions, not cards, and they renumber mid-loop when a card changes mode, so cards are addressed by BDF.
• On ml-serve1014, amd-smi set -C DPX fails because its parser builds the list of valid modes from a string and splits it into single characters. Writing current_compute_partition in sysfs works on both nodes. The amd-smi-lib version is the same on 1013 and 1014, and I haven't found why the two nodes behave differently.

Puppet

3b9ec9eea1 adds profile::amd_gpu::gpu_compute_partition_mode and reworks amd-smi-gpu-partition:

• the memory mode is staged and the driver reloaded only if the memory mode differs, so a normal boot skips the reload; the compute mode is set after any reload;
• the unit waits until the KFD topology shows every partition before the device plugin starts. The plugin enumerates GPUs only once, at startup.

On a reboot of ml-serve1013 the unit restored DPX in about 3 seconds with no driver reload.

Nodes also carry amd.com/gpu.compute-partition next to amd.com/gpu.memory-partition.