Change Details

As an engineer, I want to deploy a 7b parameter model from HuggingFace on our MI100 AMD GPU on ml-staging, so that I can identify potential challenges and bottlenecks on deploying such models. As part of this task I will deploy [[ https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 | Mixtral-8x7B ]] which is one of the [[ https://mistral.ai/news/mixtral-of-experts/ | latest open source ]] models available under an Apache 2.0 license. After successfully deploying the model we want to record inference latency for the "vanilla" version and then incrementally experiment with inference optimization techniques like 8bit/4bit quantization and Flash Attention 2 to verify that these can be used on our GPU and record improvements.

As an engineer, I want to deploy a 7b parameter model from HuggingFace on our MI100 AMD GPU on ml-staging, so that I can identify potential challenges and bottlenecks on deploying such models. As part of this task I will deploy [[ https://huggingface.co/tiiuae/falcon-7b | falcon-7b ]] which is one of the open source models available under an Apache 2.0 license. After successfully deploying the model we want to record inference latency for the "vanilla" version and then incrementally experiment with inference optimization techniques like 8bit/4bit quantization and Flash Attention 2 to verify that these can be used on our GPU and record improvements.

As an engineer, I want to deploy a 7b parameter model from HuggingFace on our MI100 AMD GPU on ml-staging, so that I can identify potential challenges and bottlenecks on deploying such models. As part of this task I will deploy [[ https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 | Mixtral-8x7B co/tiiuae/falcon-7b | falcon-7b ]] which is one of the [[ https://mistral.ai/news/mixtral-of-experts/ | latest open source ]] models available under an Apache 2.0 license. After successfully deploying the model we want to record inference latency for the "vanilla" version and then incrementally experiment with inference optimization techniques like 8bit/4bit quantization and Flash Attention 2 to verify that these can be used on our GPU and record improvements.