Summary
Deploy Qwen 3.6-27B (official FP8 quantized weights, ~30 GB) on Lift Wing using vLLM AsyncLLMEngine with tensor parallelism across 2 GPUs, supporting configurable reasoning and non-reasoning modes per request.
Technical notes
Model: Qwen/Qwen3.6-27B-FP8 (Apache 2.0), 27B dense parameters, hybrid attention (Gated DeltaNet + Gated Attention), 262K native context. Official fine-grained FP8 quantization (block size 128). Text-only serving with vision encoder disabled. Same FP8 model used everywhere.
Serving: vLLM AsyncLLMEngine via the same KServe + Blubber pattern as gpt-oss-safeguard-20b. Initial base image is amd-vllm014 (vLLM 0.14); if FP8 support is absent, a new base image with vLLM >= 0.19.0 will be needed. Reasoning mode toggled via reasoning field in the request payload. GPU_MEMORY_UTILIZATION set to 0.85, MAX_MODEL_LEN to 32768
(conservative).
Deployment: Model got deployed on 1x 192GM VRAM GPU partition of MI300X. It is deployed in llm namespace under the following name: llm-qwen36-27b. It's using FP8 quantization.
Acceptance criteria
- Model server loads Qwen/Qwen3.6-27B-FP8 and serves predictions via vLLM AsyncLLMEngine
- Upload model to swift -> s3://wmf-ml-models/llm/Qwen3.6-27B-FP8
- CI pipeline publishes the machinelearning-liftwing-inference-services-qwen36 image
- Service deployed in experimental namespace on ml-serve-eqiad and verified with curl