In this epic we should work on GPUs in the context of the ML infrastructure.
These are the main subjects to work on:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T333462 Experiment with GPUs in the Machine Learning infrastructure | |||
| Resolved | elukey | T327923 Investigate procuring and installing GPUs on Lift Wing | |||
| Resolved | elukey | T333009 Review and test the AMD GPU kubernetes plugin | |||
| Resolved | elukey | T295661 Upgrade ROCm to 5.4 | |||
| Declined | None | T275896 Review ROCm deployment procedures and current packages | |||
| Open | Spike | None | T334583 [Spike] Run models and frameworks on AMD GPU and identify challenges | ||
| Resolved | Jclark-ctr | T335031 Move two GPUs from Hadoop to Lift Wing | |||
| Open | None | T335480 Test KServe inference batching |
We have successfully deployed bloom-560m with and without GPU on LiftWing 🎉
Preliminary results show an out-of-the-box (without additional inference optimization) of 7x-10x reduced latencies. For example for a requested result length of 200 tokens the cpu version takes ~45seconds while th gpu one 4s.
A more detailed comparison will follow.
bloom-3b works as well! Tried to generate 100 tokens: with GPU ~7s, without 1:30 mins :D