Summary
As an MLE running LLM services on the MI300X nodes, I want a single Grafana dashboard that shows the key serving metrics for each model — time to first token (TTFT), time per
output token (TPOT), throughput (tokens/s), request rate — alongside GPU usage, filterable by model, so I can monitor health and debug latency or throughput issues.
Technical notes
All the underlying metrics already exist in Thanos (vLLM serving metrics and GPU metrics), so this is mainly about building the dashboard and its filters. Showing GPU usage per
model depends on the related correlation task.
Acceptance criteria
- Dashboard shows per-model serving metrics: TTFT, TPOT, throughput, and request rate.
- Dashboard shows GPU usage (utilisation, VRAM, power, temperature), filterable by model.
- Dashboard is saved and linked from the wikitech AMD GPU page.