This Epic tracks the setup of MI300X GPU nodes (ml-serve1012, ml-serve1013, ml-serve1014, ml-serve1014, ml-serve1015) as Kubernetes worker nodes.

1. Kubernetes Worker Setup
   • Add hosts as k8s worker nodes
   • Startup tweaks required for the hosts
   • Setup partitioning scheme and make sure it is used
   • Add proper labelling for the nodes and partitions
2. Network & DNS
   • Add DNS entries for routing request to eqiad T422253: Transparent DNS Routing for LiftWing Services (eqiad vs Multi-DC)
3. ROCm Integration
   • Validate vllm/ROCm Docker images work on the new nodes
   • Verify GPU device accessible from containers
4. Monitoring
   • Update GPU Grafana dashboard to be able to filter partitions based on the specific deployment/model
   • Set up and validate proper scraping of prometheus metrics available by vllm for GPU usage. There is also a great example for prometheus + grafana. Example of metrics we would like to have in the dashboard include: TTFT (Time to First Token) , TPOT (Time Per Output Token), Throughput - Tokens/second aggregate, Request Rate - Requests per second
5. Update Documentation
   • Update the documentation page on https://wikitech.wikimedia.org/wiki/Machine_Learning/AMD_GPU to reflect current host status as well as partitioning scheme.