Page MenuHomePhabricator
Create Task
Maniphest T415627

Update WMF Debian vLLM image to support latest upstream software stack
Closed, ResolvedPublic
Actions

Description

In T385173, we ported the Ubuntu vLLM docker image (Dockerfile.rocm_base, Dockerfile.rocm) and built a WMF Debian vLLM image (Dockerfile.template) that supports: ROCm 6.3.0, PyTorch 2.10.0.dev20250926+rocm6.3, CK FlashAttention 2.7.4, and vLLM 0.8.5.

The current WMF Debian vLLM image is based on an April 2025 snapshot of the upstream image. To ensure the team can leverage the latest inference optimizations, memory management features, and kernel improvements, we need to port the latest January 2026 upstream Ubuntu image (Dockerfile.rocm_base, Dockerfile.rocm) to our Debian image.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
ml: chunk torch libs in vLLM 0.14 imageKevin Baziraoperations/docker-images/production-imagesmaster+45 -2
ml: correct minor version of vLLM 0.14 imageKevin Baziraoperations/docker-images/production-imagesmaster+1 -1
ml: add vLLM 0.14 imageKevin Baziraoperations/docker-images/production-imagesmaster+162 -0
aptrepo: add ROCm 7.0.0 packages to wikimedia bookworm mirrorKevin Baziraoperations/puppetproduction+11 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

kevinbazira created this task.Jan 27 2026, 6:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2026, 6:50 AM
kevinbazira moved this task from Unsorted to In Progress on the Machine-Learning-Team board.Jan 27 2026, 6:57 AM
kevinbazira added a project: Essential-Work.
kevinbazira added a comment.EditedJan 27 2026, 7:42 AM

The Ubuntu vLLM docker image now uses ROCm 7.0.0 which supports both GPU architectures used on LiftWing: MI210(gfx90a) and MI300X(gfx942)

The Wikimedia bookworm mirror currently contains ROCm 6.3.0 as the latest packages. We need to add ROCm 7.0.0 packages on the Wikimedia bookworm mirror in order to upgrade the WMF Debian vLLM image.

Wikimedia bookworm mirror with ROCm 6.3.0 - Screenshot from 2026-01-27 10-17-29.png (1,457×826 px, 192 KB)

gerritbot added a comment.Jan 27 2026, 12:05 PM

Change #1233681 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/puppet@production] aptrepo: add ROCm 7.0.0 packages to wikimedia bookworm mirror

https://gerrit.wikimedia.org/r/1233681

gerritbot added a project: Patch-For-Review.Jan 27 2026, 12:05 PM
gerritbot added a comment.Jan 27 2026, 12:18 PM

Change #1233681 merged by Dpogorzelski:

[operations/puppet@production] aptrepo: add ROCm 7.0.0 packages to wikimedia bookworm mirror

https://gerrit.wikimedia.org/r/1233681

Maintenance_bot removed a project: Patch-For-Review.Jan 27 2026, 12:32 PM
kevinbazira added a comment.Feb 2 2026, 10:05 AM

Following T415627#11556643, we added ROCm 7.0.0 packages to the Wikimedia bookworm mirror as detailed in P87998. Afterwards, I built the wmf-debian-vllm image using the ROCm 7.0.0 packages and observed that, previously with Rocm 6.3, the torch package was the largest at ~21GB (T385173#10822741). In ROCm 7.0.0, the torch package remains the largest but has decreased to ~12GB, as shown below.

$ du -sh /srv/venv/lib/python3.11/site-packages/*  | sort -hr | head -n 10
12G     /srv/venv/lib/python3.11/site-packages/torch
3.5G    /srv/venv/lib/python3.11/site-packages/aiter
931M    /srv/venv/lib/python3.11/site-packages/triton
370M    /srv/venv/lib/python3.11/site-packages/flash_attn_2_cuda.cpython-311-x86_64-linux-gnu.so
325M    /srv/venv/lib/python3.11/site-packages/vllm
211M    /srv/venv/lib/python3.11/site-packages/ray
188M    /srv/venv/lib/python3.11/site-packages/cupy
179M    /srv/venv/lib/python3.11/site-packages/aiter_meta
149M    /srv/venv/lib/python3.11/site-packages/pyarrow
129M    /srv/venv/lib/python3.11/site-packages/llvmlite

hipblaslt dropped from ~10GB to 3.8G, and rocblas dropped from ~3.5GB to ~650MB:

$ du -sh /srv/venv/lib/python3.11/site-packages/torch/lib/*  | sort -hr | head -n 10
3.8G    /srv/venv/lib/python3.11/site-packages/torch/lib/hipblaslt
1.3G    /srv/venv/lib/python3.11/site-packages/torch/lib/libmagma.so
910M    /srv/venv/lib/python3.11/site-packages/torch/lib/libMIOpen.so
889M    /srv/venv/lib/python3.11/site-packages/torch/lib/aotriton.images
725M    /srv/venv/lib/python3.11/site-packages/torch/lib/librocsolver.so
650M    /srv/venv/lib/python3.11/site-packages/torch/lib/rocblas
433M    /srv/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so
404M    /srv/venv/lib/python3.11/site-packages/torch/lib/libtorch_hip.so
374M    /srv/venv/lib/python3.11/site-packages/torch/lib/librocsparse.so
338M    /srv/venv/lib/python3.11/site-packages/torch/lib/librocrand.so

In T385173#10826281, we had added a step to the build image that chunks these two heavy packages in the build variant, then copies smaller chunks into separate layers in the runtime image. Now that these two packages are ~0.6x smaller, this chunking step is not required.

achou mentioned this in T412357: Install AMD GPU + torch version of ML Labs machines.Feb 3 2026, 1:09 AM
kevinbazira added a comment.Feb 3 2026, 2:26 PM

I've upgraded the wmf-debian-vllm image to support the latest upstream software stack as of Jan 2026. Below is the updated dockerfile:

P88499 dockerfile for wmf-debian-vllm image that supports latest upstream software stack as of Jan 2026
1########################################################################
2# wmf-debian-vllm: ROCm, PyTorch, MoRi, FlashAttention, aiter, vLLM #
3# #
4# Note: Multiple RUN commands are intentionally kept separate to avoid #
5# hitting the 12GB (compressed) Docker layer limit required by the #
6# Wikimedia Docker registry. #
7########################################################################
8
9FROM docker-registry.wikimedia.org/amd-pytorch-common:0.0.1-2-20260118 as builder
10
11# Set proxy env vars required on ml-lab1002 (see: https://phabricator.wikimedia.org/P75284#302759)
12ARG http_proxy
13ENV http_proxy=${http_proxy}
14ENV https_proxy=${http_proxy}
15ENV HTTP_PROXY=${http_proxy}
16ENV HTTPS_PROXY=${http_proxy}
17
18# Mirror upstream: create 'render' group
19RUN groupadd -g 109 render
20WORKDIR /srv
21
22# Add Wikimedia ROCm 7.0 mirror then install ROCm libs and Python tooling
23RUN echo "deb http://apt.wikimedia.org/wikimedia bookworm-wikimedia thirdparty/amd-rocm70" > /etc/apt/sources.list.d/rocm.list \
24 && apt-get update -q \
25 && apt-get install -q -y libopenmpi-dev libpci-dev hsa-rocr-dev miopen-hip mivisionx radeontop rccl rocblas rocfft rocm-cmake rocm-dev rocm-device-libs rocm-libs rocm-opencl rocm-opencl-dev rocm-utils rocrand rocm-smi-lib migraphx cmake build-essential python3-dev python3-venv git curl sudo vim sqlite3 libsqlite3-dev libfmt-dev libmsgpack-dev libsuitesparse-dev
26
27# Set environment for ROCm and vLLM
28ENV ROCM_PATH=/opt/rocm-7.0.0 \
29 VLLM_TARGET_DEVICE=rocm \
30 # For more details on AMD GPU architectures like gfx90a, see: https://rocm.docs.amd.com/en/latest/reference/gpu-arch-specs.html
31 PYTORCH_ROCM_ARCH=gfx90a;gfx942 \
32 PATH=/opt/rocm-7.0.0/llvm/bin:/opt/rocm-7.0.0/bin:/srv/venv/bin:$PATH \
33 # LD_LIBRARY_PATH is needed to ensure that the runtime can locate shared libraries in non-standard directories, such as /opt/rocm-7.0.0/lib and /usr/local/lib, which are required by ROCm and other dependencies.
34 LD_LIBRARY_PATH=/opt/rocm-7.0.0/lib:/usr/local/lib:
35
36# Create a Python virtual environment and a custom temp directory
37RUN python3 -m venv /srv/venv \
38 && mkdir -p /opt/tmp
39ENV PATH="/srv/venv/bin:${PATH}"
40ENV TMPDIR=/opt/tmp
41
42# Install ROCm-enabled PyTorch (into the venv)
43# Using the stable index instead of the nightly index as it's more stable. (see: https://phabricator.wikimedia.org/P87924)
44RUN pip install --no-cache-dir --pre torch==2.10.0+rocm7.0 \
45 --index-url https://download.pytorch.org/whl/rocm7.0 \
46 --extra-index-url https://pypi.org/simple
47
48# Install the AMD SMI Python interface
49RUN pip install --no-cache-dir /opt/rocm-7.0.0/share/amd_smi
50
51# Install Python build packages required by both FlashAttention and vLLM
52RUN pip install --no-cache-dir setuptools_scm packaging \
53 "cmake<4" ninja wheel "setuptools<80" pybind11 Cython
54
55# Build MoRI in a separate stage
56FROM builder AS mori-builder
57RUN git clone https://github.com/ROCm/mori.git /srv/mori \
58 && cd /srv/mori \
59 && git checkout 2d02c6a9 \
60 && git submodule update --init --recursive \
61 && GPU_TARGETS="gfx90a;gfx942" python3 setup.py bdist_wheel \
62 && mkdir -p /srv/wheels \
63 && mv dist/*.whl /srv/wheels \
64 && cd /srv \
65 && rm -rf /srv/mori
66
67# Build FlashAttention in a separate stage
68FROM builder AS flashattention-builder
69RUN git clone https://github.com/Dao-AILab/flash-attention.git /srv/flash-attn \
70 && cd /srv/flash-attn \
71 && git checkout 0e60e394 \
72 && git submodule update --init \
73 # For more details on AMD GPU architectures like gfx90a, see: https://rocm.docs.amd.com/en/latest/reference/gpu-arch-specs.html
74 && GPU_ARCHS="gfx90a;gfx942" python3 setup.py bdist_wheel \
75 && mkdir -p /srv/wheels \
76 && mv dist/*.whl /srv/wheels \
77 && cd /srv \
78 && rm -rf /srv/flash-attn
79
80# Build aiter in a separate stage
81FROM builder AS aiter-builder
82RUN git clone https://github.com/ROCm/aiter.git /srv/aiter \
83 && cd /srv/aiter \
84 && git checkout 6af8b687 \
85 && git submodule sync \
86 && git submodule update --init --recursive \
87 && pip install -r requirements.txt \
88 # Currently doesn't support gfx90a. (see: https://phabricator.wikimedia.org/P88226)
89 && PREBUILD_KERNELS=1 GPU_ARCHS=gfx942 python3 setup.py bdist_wheel \
90 && mkdir -p /srv/wheels \
91 && mv dist/*.whl /srv/wheels \
92 && cd /srv \
93 && rm -rf /srv/aiter
94
95# Build vLLM in a separate stage
96FROM builder AS vllm-builder
97RUN git clone --branch main https://github.com/vllm-project/vllm.git /srv/vllm \
98 && cd /srv/vllm \
99 && git checkout 6c0064571 \
100 && git submodule update --init \
101 && pip install --no-cache-dir -r requirements/rocm.txt \
102 && GPU_ARCHS="gfx90a;gfx942" python3 setup.py bdist_wheel \
103 && mkdir -p /srv/wheels \
104 && mv dist/*.whl /srv/wheels \
105 && cd /srv \
106 && rm -rf /srv/vllm
107
108# Final stage: Create minimal runtime image
109FROM docker-registry.wikimedia.org/amd-pytorch-common:0.0.1-2-20260118 as runtime
110
111# Set proxy env vars required on ml-lab1002 (see: https://phabricator.wikimedia.org/P75284#302759)
112ARG http_proxy
113ENV http_proxy=${http_proxy}
114ENV https_proxy=${http_proxy}
115ENV HTTP_PROXY=${http_proxy}
116ENV HTTPS_PROXY=${http_proxy}
117
118WORKDIR /srv
119
120# Set environment for ROCm and vLLM
121ENV ROCM_PATH=/opt/rocm-7.0.0 \
122 ROCM_HOME=/opt/rocm-7.0.0 \
123 VLLM_TARGET_DEVICE=rocm \
124 PATH=/opt/rocm-7.0.0/llvm/bin:/opt/rocm-7.0.0/bin:/srv/venv/bin:$PATH \
125 # LD_LIBRARY_PATH is needed to ensure that the runtime can locate shared libraries in non-standard directories, such as /opt/rocm-7.0.0/lib and /usr/local/lib, which are required by ROCm and other dependencies.
126 LD_LIBRARY_PATH=/srv/venv/lib/python3.11/site-packages/torch/lib:/opt/rocm-7.0.0/lib:/usr/local/lib:
127
128# Add Wikimedia ROCm 7.0 mirror then install minimal runtime ROCm libs and Python tooling
129RUN echo "deb http://apt.wikimedia.org/wikimedia bookworm-wikimedia thirdparty/amd-rocm70" > /etc/apt/sources.list.d/rocm.list \
130 && apt-get update -q \
131 && apt-get install -q -y libopenmpi-dev libpci-dev libc6-dev rocm-smi-lib hip-dev rccl python3-dev ca-certificates gcc \
132 # Cleanup apt cache. Not required in final WMF image as it's done automagically.
133 && apt-get clean \
134 && rm -rf /var/lib/apt/lists/*
135
136# Remove large unused static libraries
137# We only need the .so files for runtime inference.
138RUN rm -f /opt/rocm-7.0.0/lib/*.a
139
140# Copy venv
141COPY --from=builder --chown=somebody /srv/venv /srv/venv
142
143# Copy pre-built wheels for MoRI, FlashAttention, aiter, and vLLM
144COPY --from=mori-builder --chown=somebody /srv/wheels /srv/wheels
145COPY --from=flashattention-builder --chown=somebody /srv/wheels /srv/wheels
146COPY --from=aiter-builder --chown=somebody /srv/wheels /srv/wheels
147COPY --from=vllm-builder --chown=somebody /srv/wheels /srv/wheels
148
149# Switch to user "somebody" to avoid running the container as root
150USER somebody
151
152# Install Python packages from wheels and uninstall unused package torchvision
153RUN pip install --no-cache-dir /srv/wheels/*.whl \
154 && pip uninstall -y torchvision \
155 # To reduce the image size, remove the wheel files as they are no longer needed
156 && rm -rf /srv/wheels/*
157
158# Set performance environment variables as defined in upstream vLLM ROCm dockerfiles
159ENV RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES=1 \
160 RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES=1 \
161 TOKENIZERS_PARALLELISM=false \
162 # ENV that can improve safe tensor loading, and end-to-end time
163 SAFETENSORS_FAST_GPU=1 \
164 # Performance environment variable.
165 HIP_FORCE_DEV_KERNARG=1 \
166 # Required for RCCL in ROCm7.1 (kept this because it also exists in the rocm/vllm:v0.14.0_amd_dev docker image that runs ROCm7.0)
167 HSA_NO_SCRATCH_RECLAIM=1

The key updates are:

  • Upgraded packages:
    • ROCm 6.3.0 -> 7.0.0
    • torch 2.10.0.dev20250926+rocm6.3 -> 2.10.0+rocm7.0
    • flash_attn 2.7.4 -> 2.8.3
    • vLLM 0.8.5 -> 0.14
  • Added packages:
    • mori 0.1
    • aiter 0.1.7
  • Previously only targetted MI210 (gfx90a) GPUs, now targets both MI210 (gfx90a) and MI300X (gfx942) GPUs.
  • Nolonger using the nightly index for torch installation as it's unstable. Using the stable index instead. (P87924)
  • Removed chunking of torch libs (hipblaslt and rocblas) because T415627#11573039
  • Added performance environment variables in runtime image as defined upstream.

The uncompressed image layer sizes are shown below with the largest being ~13.2GB:

$ docker history wmf-vllm-image:upgrade
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
62ba3f1cbf88   4 minutes ago   /bin/sh -c #(nop)  ENV RAY_EXPERIMENTAL_NOSE…   0B        
5be2f086254a   5 minutes ago   /bin/sh -c pip install --no-cache-dir /srv/w…   6.92GB    
771973b901b1   2 hours ago     /bin/sh -c #(nop)  USER somebody                0B        
ac95c2abcbca   2 hours ago     /bin/sh -c #(nop) COPY --chown=somebodydir:a…   71.1MB    
65c3a17e79d4   2 hours ago     /bin/sh -c #(nop) COPY --chown=somebodydir:a…   478MB     
af0f97b6f155   2 hours ago     /bin/sh -c #(nop) COPY --chown=somebodydir:0…   96.9MB    
248ca720ebec   2 hours ago     /bin/sh -c #(nop) COPY --chown=somebodydir:c…   1.24MB    
5f51dc35e350   2 hours ago     /bin/sh -c #(nop) COPY --chown=somebodydir:2…   13.2GB    
bbb849f47fbd   2 hours ago     /bin/sh -c rm -f /opt/rocm-7.0.0/lib/*.a        0B        
f82b5d9eced2   2 hours ago     /bin/sh -c echo "deb http://apt.wikimedia.or…   3.11GB    
3585b6f23e1e   2 hours ago     /bin/sh -c #(nop)  ENV ROCM_PATH=/opt/rocm-7…   0B        
10e57b485275   2 hours ago     /bin/sh -c #(nop) WORKDIR /srv                  0B        
8ceb8fe625b3   6 hours ago     /bin/sh -c #(nop)  ENV HTTPS_PROXY=http://we…   0B        
ed64d9b168ba   6 hours ago     /bin/sh -c #(nop)  ENV HTTP_PROXY=http://web…   0B        
8cbe5721e4d0   6 hours ago     /bin/sh -c #(nop)  ENV https_proxy=http://we…   0B        
8fa827f7a01f   6 hours ago     /bin/sh -c #(nop)  ENV http_proxy=http://web…   0B        
9b38100fd4e1   6 hours ago     /bin/sh -c #(nop)  ARG http_proxy               0B        
6772f2ce6f63   2 weeks ago     |0 /bin/sh -c echo 'Acquire::http::Proxy "ht…   1.53MB    
<missing>      2 weeks ago     |0 /bin/sh -c echo 'Acquire::http::Proxy "ht…   75.5MB    
<missing>      2 weeks ago     /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B        
<missing>      2 weeks ago     /bin/sh -c #(nop)  ENV LC_ALL=C.UTF-8           0B        
<missing>      2 weeks ago     /bin/sh -c #(nop) ADD file:fd31ad82b8214f5d0…   74.8MB

The compressed layer sizes on dockerhub show the largest is ~4.88GB, which meets the wikimedia docker registry 12GB compressed layer size limit.

kevinbazira added a comment.Feb 3 2026, 2:46 PM

I have tested the image built in T415627#11578754 on ML-Lab. It was able to successfully load and run inference on the aya-expanse-8b model using an MI210 GPU.

P88502 vllm serving aya-expanse-8b in wmf-debian-vllm container that supports ROCm 7.0.0 and vLLM 0.14 on MI210 GPU
1$ docker run --rm --network=host -it -e VLLM_USE_TRITON_FLASH_ATTN=0 -e HF_TOKEN=<redacted> --device=/dev/kfd --device=/dev/dri --group-add=$(getent group video | cut -d: -f3) --group-add=$(getent group render | cut -d: -f3) --ipc=host --security-opt seccomp=unconfined -v /srv/hf-cache:/home/vllm/.cache/huggingface wmf-vllm-image:upgrade_1 /srv/venv/bin/python -c "
2from vllm import LLM, SamplingParams; \
3llm = LLM('CohereForAI/aya-expanse-8b'); \
4print(llm.generate('Hello, vLLM!', SamplingParams(max_tokens=10))[0].outputs[0].text)"
5/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7INFO 02-03 14:40:30 [utils.py:282] non-default args: {'disable_log_stats': True, 'model': 'CohereForAI/aya-expanse-8b'}
8config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 634/634 [00:00<00:00, 3.43MB/s]
9[aiter] import [module_aiter_enum] under /srv/venv/lib/python3.11/site-packages/aiter/jit/module_aiter_enum.so
10[2026-02-03 14:40:40] INFO core.py:477: import [module_aiter_enum] under /srv/venv/lib/python3.11/site-packages/aiter/jit/module_aiter_enum.so
11INFO 02-03 14:40:40 [model.py:541] Resolved architecture: CohereForCausalLM
12INFO 02-03 14:40:40 [model.py:1561] Using max model len 8192
13INFO 02-03 14:40:41 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=8192.
14INFO 02-03 14:40:41 [vllm.py:624] Asynchronous scheduling is enabled.
15generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 137/137 [00:00<00:00, 1.11MB/s]
16tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 8.64k/8.64k [00:00<00:00, 37.2MB/s]
17tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12.8M/12.8M [00:00<00:00, 26.6MB/s]
18special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 3.11MB/s]
19WARNING 02-03 14:40:45 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
20/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
21/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
22(EngineCore_DP0 pid=316) INFO 02-03 14:40:49 [core.py:96] Initializing a V1 LLM engine (v0.14.0rc2.dev374+g6c0064571) with config: model='CohereForAI/aya-expanse-8b', speculative_config=None, tokenizer='CohereForAI/aya-expanse-8b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=CohereForAI/aya-expanse-8b, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none', '+sparse_attn_indexer'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None}
23(EngineCore_DP0 pid=316) [aiter] import [module_aiter_enum] under /srv/venv/lib/python3.11/site-packages/aiter/jit/module_aiter_enum.so
24(EngineCore_DP0 pid=316) [2026-02-03 14:40:50] INFO core.py:477: import [module_aiter_enum] under /srv/venv/lib/python3.11/site-packages/aiter/jit/module_aiter_enum.so
25(EngineCore_DP0 pid=316) INFO 02-03 14:40:51 [parallel_state.py:1234] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.64.161.5:55539 backend=nccl
26(EngineCore_DP0 pid=316) INFO 02-03 14:40:51 [parallel_state.py:1445] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
27(EngineCore_DP0 pid=316) INFO 02-03 14:40:51 [gpu_model_runner.py:4021] Starting to load model CohereForAI/aya-expanse-8b...
28(EngineCore_DP0 pid=316) INFO 02-03 14:40:52 [rocm.py:338] Using Triton Attention backend.
29(EngineCore_DP0 pid=316) WARNING 02-03 14:40:52 [compilation.py:1048] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
30model.safetensors.index.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 21.0k/21.0k [00:00<00:00, 73.2MB/s]
31model-00004-of-00004.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 1.22G/1.22G [00:13<00:00, 92.6MB/s]
32model-00002-of-00004.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 4.92G/4.92G [00:22<00:00, 215MB/s]
33model-00003-of-00004.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 5.00G/5.00G [00:23<00:00, 217MB/s]
34model-00001-of-00004.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 4.92G/4.92G [00:23<00:00, 211MB/s]
35(EngineCore_DP0 pid=316) INFO 02-03 14:41:17 [weight_utils.py:527] Time spent downloading weights for CohereForAI/aya-expanse-8b: 23.660938 seconds:23<00:01, 787MB/s]
36Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]████████████████████████████████████████████████ | 4.65G/4.92G [00:23<00:00, 1.53GB/s]
37Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:02, 1.45it/s]
38Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:03<00:03, 1.78s/it]
39Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:06<00:02, 2.29s/it]
40Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00, 2.51s/it]
41Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:08<00:00, 2.25s/it]
42(EngineCore_DP0 pid=316)
43(EngineCore_DP0 pid=316) INFO 02-03 14:41:26 [default_loader.py:291] Loading weights took 9.35 seconds
44(EngineCore_DP0 pid=316) INFO 02-03 14:41:27 [gpu_model_runner.py:4118] Model loading took 15.01 GiB memory and 34.616557 seconds
45(EngineCore_DP0 pid=316) INFO 02-03 14:42:13 [backends.py:805] Using cache directory: /home/somebody/.cache/vllm/torch_compile_cache/add1060b6e/rank_0_0/backbone for vLLM's torch.compile
46(EngineCore_DP0 pid=316) INFO 02-03 14:42:13 [backends.py:865] Dynamo bytecode transform time: 46.52 s
47(EngineCore_DP0 pid=316) INFO 02-03 14:42:24 [backends.py:302] Cache the graph of compile range (1, 8192) for later use
48(EngineCore_DP0 pid=316) INFO 02-03 14:42:42 [backends.py:319] Compiling a graph for compile range (1, 8192) takes 26.51 s
49(EngineCore_DP0 pid=316) INFO 02-03 14:42:42 [monitor.py:34] torch.compile takes 73.04 s in total
50(EngineCore_DP0 pid=316) INFO 02-03 14:42:42 [decorators.py:576] saving AOT compiled function to /home/somebody/.cache/vllm/torch_aot_compile/633a9a29e52ef4288cef02a54e62bd4ae40a6c42aac658f0eb443fb19d1183a2/rank_0_0/model
51(EngineCore_DP0 pid=316) INFO 02-03 14:42:43 [decorators.py:580] saved AOT compiled function to /home/somebody/.cache/vllm/torch_aot_compile/633a9a29e52ef4288cef02a54e62bd4ae40a6c42aac658f0eb443fb19d1183a2/rank_0_0/model
52(EngineCore_DP0 pid=316) INFO 02-03 14:42:45 [gpu_worker.py:356] Available KV cache memory: 39.72 GiB
53(EngineCore_DP0 pid=316) INFO 02-03 14:42:45 [kv_cache_utils.py:1307] GPU KV cache size: 325,392 tokens
54(EngineCore_DP0 pid=316) INFO 02-03 14:42:45 [kv_cache_utils.py:1312] Maximum concurrency for 8,192 tokens per request: 39.72x
55Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|████████████████████████████████████████████████████████████████████████| 51/51 [00:03<00:00, 15.60it/s]
56Capturing CUDA graphs (decode, FULL): 100%|███████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:04<00:00, 7.24it/s]
57(EngineCore_DP0 pid=316) INFO 02-03 14:42:53 [gpu_model_runner.py:5051] Graph capturing finished in 9 secs, took 0.44 GiB
58(EngineCore_DP0 pid=316) INFO 02-03 14:42:53 [core.py:272] init engine (profile, create kv cache, warmup model) took 86.64 seconds
59(EngineCore_DP0 pid=316) INFO 02-03 14:42:55 [vllm.py:624] Asynchronous scheduling is enabled.
60INFO 02-03 14:42:55 [llm.py:343] Supported tasks: ['generate']
61Adding requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 389.23it/s]
62Processed prompts: 100%|██████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.10it/s, est. speed input: 14.71 toks/s, output: 21.01 toks/s]
63
64The hardest part about making and releasing a free

gerritbot added a comment.Feb 5 2026, 7:41 AM

Change #1237060 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/docker-images/production-images@master] ml: add vLLM 0.14 image

https://gerrit.wikimedia.org/r/1237060

gerritbot added a project: Patch-For-Review.Feb 5 2026, 7:41 AM
gerritbot added a comment.Feb 6 2026, 8:40 AM

Change #1237060 merged by Dpogorzelski:

[operations/docker-images/production-images@master] ml: add vLLM 0.14 image

https://gerrit.wikimedia.org/r/1237060

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2026, 9:30 AM
kevinbazira added a subscriber: DPogorzelski-WMF.Feb 9 2026, 4:15 AM

@DPogorzelski-WMF tried pushing the new image to the the wikimedia docker registry:

root@ml-build1001:/home/dpogorzelski# docker push docker-registry.discovery.wmnet/ml/amd-vllm014:gfx90agfx942rocm7.0.0pytorch2.10.0mori0.1flash-attn2.8.3aiter0.1.7vllm0.14
The push refers to repository [docker-registry.discovery.wmnet/ml/amd-vllm014]
daff26c4c2ca: Layer already exists
4e6fe3bb1cef: Layer already exists
134fa4016583: Layer already exists
af738ff63f36: Layer already exists
849c6b8e4c58: Layer already exists
5d360044330f: Pushing [==========================>                        ]  6.989GB/13.19GB
5d360044330f: Pushing [==========================>                        ]  7.059GB/13.19GB
5d360044330f: Pushing [==========================>                        ]  7.066GB/13.19GB
5d360044330f: Pushing [==========================>                        ]  7.102GB/13.19GB
5d360044330f: Pushing [===========================>                       ]  7.123GB/13.19GB
5d360044330f: Pushing [===========================>                       ]  7.132GB/13.19GB
5d360044330f: Pushing [===========================>                       ]  7.136GB/13.19GB
5d360044330f: Pushing [==================================>                ]  9.088GB/13.19GB
5d360044330f: Pushing [=====================================>             ]  9.926GB/13.19GB
5d360044330f: Retrying in 1 second

5d360044330f: Pushing [>                                                  ]  87.97MB/13.19GB

but it kept failing with the error below:

Feb 06 14:22:18 ml-build1001 dockerd[726938]: time="2026-02-06T14:22:18.883703814Z" level=error msg="Upload failed, retrying: unknown: <html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\>

This error indicates that there is an image layer being pushed that is larger than what the wikimedia docker registry is configured to accept.

For compressed layer sizes, 4GB was the limit 11 months ago, I see it was bumped to 12GB recently. In T415627#11578754 we saw that the largest compressed layer size of the new image is ~4.88GB, which is larger than the previous 4GB limit but smaller than the new 12GB limit.

To unblock ourselves, we have reintroduced chunking of the torch libs (hipblaslt and rocblas), and the largest uncompressed layer size has dropped from ~13.2GB to ~8.54GB:

$ docker history wmf-vllm-image:upgrade
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
94dfdca93de3   40 minutes ago   /bin/sh -c #(nop)  ENV RAY_EXPERIMENTAL_NOSE…   0B        
ad7e00e56ca4   40 minutes ago   |1 TORCH_LIB_PATH=/srv/venv/lib/python3.11/s…   6.92GB    
d6bee6c55195   45 minutes ago   /bin/sh -c #(nop)  USER somebody                0B        
cf6302663bf9   45 minutes ago   /bin/sh -c #(nop) COPY --chown=somebodydir:4…   71.1MB    
113cd7b3cd8b   45 minutes ago   /bin/sh -c #(nop) COPY --chown=somebodydir:6…   478MB     
3716e126a209   45 minutes ago   /bin/sh -c #(nop) COPY --chown=somebodydir:0…   96.9MB    
e380824fd989   46 minutes ago   /bin/sh -c #(nop) COPY --chown=somebodydir:7…   1.24MB    
781cbf04e4b6   46 minutes ago   /bin/sh -c #(nop) COPY --chown=somebodydir:4…   677MB     
6f2b6e9ab8b1   46 minutes ago   /bin/sh -c #(nop) COPY --chown=somebodydir:4…   3.97GB    
329a0a4e9588   47 minutes ago   /bin/sh -c #(nop)  ARG TORCH_LIB_PATH=/srv/v…   0B        
6f24a1788743   47 minutes ago   /bin/sh -c #(nop) COPY --chown=somebodydir:4…   8.54GB    
cedbe0a5490e   50 minutes ago   /bin/sh -c rm -f /opt/rocm-7.0.0/lib/*.a        0B        
82e23fd731f6   50 minutes ago   /bin/sh -c echo "deb http://apt.wikimedia.or…   3.11GB    
413e47393ddd   51 minutes ago   /bin/sh -c #(nop)  ENV ROCM_PATH=/opt/rocm-7…   0B        
f9a5daa02ff6   51 minutes ago   /bin/sh -c #(nop) WORKDIR /srv                  0B        
9f1e8f23abb1   5 hours ago      /bin/sh -c #(nop)  ENV HTTPS_PROXY=http://we…   0B        
4d1a3123e3b4   5 hours ago      /bin/sh -c #(nop)  ENV HTTP_PROXY=http://web…   0B        
8169eacfd982   5 hours ago      /bin/sh -c #(nop)  ENV https_proxy=http://we…   0B        
546aa4641c2c   5 hours ago      /bin/sh -c #(nop)  ENV http_proxy=http://web…   0B        
f661fe7e2901   5 hours ago      /bin/sh -c #(nop)  ARG http_proxy               0B        
6772f2ce6f63   2 weeks ago      |0 /bin/sh -c echo 'Acquire::http::Proxy "ht…   1.53MB    
<missing>      2 weeks ago      |0 /bin/sh -c echo 'Acquire::http::Proxy "ht…   75.5MB    
<missing>      2 weeks ago      /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B        
<missing>      2 weeks ago      /bin/sh -c #(nop)  ENV LC_ALL=C.UTF-8           0B        
<missing>      2 weeks ago      /bin/sh -c #(nop) ADD file:fd31ad82b8214f5d0…   74.8MB

The largest compressed layer size on dockerhub is now ~3.69GB, which is smaller than both registry limits mentioned above.

Below is the updated dockerfile:

P88720 dockerfile for wmf-debian-vllm image that supports vLLM 0.14 with torch-libs-chunker to reduce layer sizes
1########################################################################
2# wmf-debian-vllm: ROCm, PyTorch, MoRi, FlashAttention, aiter, vLLM #
3# #
4# Note: Multiple RUN commands are intentionally kept separate to avoid #
5# hitting the 12GB (compressed) Docker layer limit required by the #
6# Wikimedia Docker registry. #
7########################################################################
8
9FROM docker-registry.wikimedia.org/amd-pytorch-common:0.0.1-2-20260118 as builder
10
11# Set proxy env vars required on ml-lab1002 (see: https://phabricator.wikimedia.org/P75284#302759)
12ARG http_proxy
13ENV http_proxy=${http_proxy}
14ENV https_proxy=${http_proxy}
15ENV HTTP_PROXY=${http_proxy}
16ENV HTTPS_PROXY=${http_proxy}
17
18# Mirror upstream: create 'render' group
19RUN groupadd -g 109 render
20WORKDIR /srv
21
22# Add Wikimedia ROCm 7.0 mirror then install ROCm libs and Python tooling
23RUN echo "deb http://apt.wikimedia.org/wikimedia bookworm-wikimedia thirdparty/amd-rocm70" > /etc/apt/sources.list.d/rocm.list \
24 && apt-get update -q \
25 && apt-get install -q -y libopenmpi-dev libpci-dev hsa-rocr-dev miopen-hip mivisionx radeontop rccl rocblas rocfft rocm-cmake rocm-dev rocm-device-libs rocm-libs rocm-opencl rocm-opencl-dev rocm-utils rocrand rocm-smi-lib migraphx cmake build-essential python3-dev python3-venv git curl sudo vim sqlite3 libsqlite3-dev libfmt-dev libmsgpack-dev libsuitesparse-dev
26
27# Set environment for ROCm and vLLM
28ENV ROCM_PATH=/opt/rocm-7.0.0 \
29 VLLM_TARGET_DEVICE=rocm \
30 # For more details on AMD GPU architectures like gfx90a, see: https://rocm.docs.amd.com/en/latest/reference/gpu-arch-specs.html
31 PYTORCH_ROCM_ARCH=gfx90a;gfx942 \
32 PATH=/opt/rocm-7.0.0/llvm/bin:/opt/rocm-7.0.0/bin:/srv/venv/bin:$PATH \
33 # LD_LIBRARY_PATH is needed to ensure that the runtime can locate shared libraries in non-standard directories, such as /opt/rocm-7.0.0/lib and /usr/local/lib, which are required by ROCm and other dependencies.
34 LD_LIBRARY_PATH=/opt/rocm-7.0.0/lib:/usr/local/lib:
35
36# Create a Python virtual environment and a custom temp directory
37RUN python3 -m venv /srv/venv \
38 && mkdir -p /opt/tmp
39ENV PATH="/srv/venv/bin:${PATH}"
40ENV TMPDIR=/opt/tmp
41
42# Install ROCm-enabled PyTorch (into the venv)
43# Using the stable index instead of the nightly index as it's more stable. (see: https://phabricator.wikimedia.org/P87924)
44RUN pip install --no-cache-dir --pre torch==2.10.0+rocm7.0 \
45 --index-url https://download.pytorch.org/whl/rocm7.0 \
46 --extra-index-url https://pypi.org/simple
47
48# Install the AMD SMI Python interface
49RUN pip install --no-cache-dir /opt/rocm-7.0.0/share/amd_smi
50
51# Install Python build packages required by both FlashAttention and vLLM
52RUN pip install --no-cache-dir setuptools_scm packaging \
53 "cmake<4" ninja wheel "setuptools<80" pybind11 Cython
54
55# Chunck torch libs (hipblaslt and rocblas) because they are large.
56# The goal is to copy them separately in the runtime stage to avoid hitting
57# the compressed layer limit in the Wikimedia Docker registry. (T415627#11594815)
58FROM builder AS torch-libs-chunker
59# Chunk entire hipblaslt and rocblas directories from PyTorch installation
60# Define paths for clarity. Using Python 3.11 from bookworm base.
61ENV TORCH_LIB_PATH="/srv/venv/lib/python3.11/site-packages/torch/lib"
62ENV HIPBLASLT_FULL_PATH="${TORCH_LIB_PATH}/hipblaslt"
63ENV ROCBLAS_FULL_PATH="${TORCH_LIB_PATH}/rocblas"
64
65# Create parent directory for the torch lib chunks
66RUN mkdir -p /srv/torch_lib_chunks
67
68# Move the entire hipblaslt directory to the chunk location
69RUN if [ -d "${HIPBLASLT_FULL_PATH}" ]; then \
70 mv "${HIPBLASLT_FULL_PATH}" /srv/torch_lib_chunks/hipblaslt && \
71 echo "Moved ${HIPBLASLT_FULL_PATH} to /srv/torch_lib_chunks/hipblaslt" ; \
72 else \
73 echo "Warning: Directory ${HIPBLASLT_FULL_PATH} not found." ; \
74 fi
75
76# Move the entire rocblas directory to the chunk location
77RUN if [ -d "${ROCBLAS_FULL_PATH}" ]; then \
78 mv "${ROCBLAS_FULL_PATH}" /srv/torch_lib_chunks/rocblas && \
79 echo "Moved ${ROCBLAS_FULL_PATH} to /srv/torch_lib_chunks/rocblas" ; \
80 else \
81 echo "Warning: Directory ${ROCBLAS_FULL_PATH} not found." ; \
82 fi
83
84# Build MoRI in a separate stage
85FROM builder AS mori-builder
86RUN git clone https://github.com/ROCm/mori.git /srv/mori \
87 && cd /srv/mori \
88 && git checkout 2d02c6a9 \
89 && git submodule update --init --recursive \
90 && GPU_TARGETS="gfx90a;gfx942" python3 setup.py bdist_wheel \
91 && mkdir -p /srv/wheels \
92 && mv dist/*.whl /srv/wheels \
93 && cd /srv \
94 && rm -rf /srv/mori
95
96# Build FlashAttention in a separate stage
97FROM builder AS flashattention-builder
98RUN git clone https://github.com/Dao-AILab/flash-attention.git /srv/flash-attn \
99 && cd /srv/flash-attn \
100 && git checkout 0e60e394 \
101 && git submodule update --init \
102 # For more details on AMD GPU architectures like gfx90a, see: https://rocm.docs.amd.com/en/latest/reference/gpu-arch-specs.html
103 && GPU_ARCHS="gfx90a;gfx942" python3 setup.py bdist_wheel \
104 && mkdir -p /srv/wheels \
105 && mv dist/*.whl /srv/wheels \
106 && cd /srv \
107 && rm -rf /srv/flash-attn
108
109# Build aiter in a separate stage
110FROM builder AS aiter-builder
111RUN git clone https://github.com/ROCm/aiter.git /srv/aiter \
112 && cd /srv/aiter \
113 && git checkout 6af8b687 \
114 && git submodule sync \
115 && git submodule update --init --recursive \
116 && pip install -r requirements.txt \
117 # Currently doesn't support gfx90a. (see: https://phabricator.wikimedia.org/P88226)
118 && PREBUILD_KERNELS=1 GPU_ARCHS=gfx942 python3 setup.py bdist_wheel \
119 && mkdir -p /srv/wheels \
120 && mv dist/*.whl /srv/wheels \
121 && cd /srv \
122 && rm -rf /srv/aiter
123
124# Build vLLM in a separate stage
125FROM builder AS vllm-builder
126RUN git clone --branch main https://github.com/vllm-project/vllm.git /srv/vllm \
127 && cd /srv/vllm \
128 && git checkout 6c0064571 \
129 && git submodule update --init \
130 && pip install --no-cache-dir -r requirements/rocm.txt \
131 && GPU_ARCHS="gfx90a;gfx942" python3 setup.py bdist_wheel \
132 && mkdir -p /srv/wheels \
133 && mv dist/*.whl /srv/wheels \
134 && cd /srv \
135 && rm -rf /srv/vllm
136
137# Final stage: Create minimal runtime image
138FROM docker-registry.wikimedia.org/amd-pytorch-common:0.0.1-2-20260118 as runtime
139
140# Set proxy env vars required on ml-lab1002 (see: https://phabricator.wikimedia.org/P75284#302759)
141ARG http_proxy
142ENV http_proxy=${http_proxy}
143ENV https_proxy=${http_proxy}
144ENV HTTP_PROXY=${http_proxy}
145ENV HTTPS_PROXY=${http_proxy}
146
147WORKDIR /srv
148
149# Set environment for ROCm and vLLM
150ENV ROCM_PATH=/opt/rocm-7.0.0 \
151 ROCM_HOME=/opt/rocm-7.0.0 \
152 VLLM_TARGET_DEVICE=rocm \
153 PATH=/opt/rocm-7.0.0/llvm/bin:/opt/rocm-7.0.0/bin:/srv/venv/bin:$PATH \
154 # LD_LIBRARY_PATH is needed to ensure that the runtime can locate shared libraries in non-standard directories, such as /opt/rocm-7.0.0/lib and /usr/local/lib, which are required by ROCm and other dependencies.
155 LD_LIBRARY_PATH=/srv/venv/lib/python3.11/site-packages/torch/lib:/opt/rocm-7.0.0/lib:/usr/local/lib:
156
157# Add Wikimedia ROCm 7.0 mirror then install minimal runtime ROCm libs and Python tooling
158RUN echo "deb http://apt.wikimedia.org/wikimedia bookworm-wikimedia thirdparty/amd-rocm70" > /etc/apt/sources.list.d/rocm.list \
159 && apt-get update -q \
160 && apt-get install -q -y libopenmpi-dev libpci-dev libc6-dev rocm-smi-lib hip-dev rccl python3-dev ca-certificates gcc \
161 # Cleanup apt cache. Not required in final WMF image as it's done automagically.
162 && apt-get clean \
163 && rm -rf /var/lib/apt/lists/*
164
165# Remove large unused static libraries
166# We only need the .so files for runtime inference.
167RUN rm -f /opt/rocm-7.0.0/lib/*.a
168
169# Copy venv (main structure; torch/lib will be missing hipblaslt and rocblas because they were moved in torch-libs-chunker stage)
170COPY --from=torch-libs-chunker --chown=somebody /srv/venv /srv/venv
171
172# Define runtime paths corresponding to torch-libs-chunker paths. Using builder's Python 3.11 from bookworm base.
173# This directory should typically exist after copying /srv/venv.
174ARG TORCH_LIB_PATH="/srv/venv/lib/python3.11/site-packages/torch/lib"
175
176# Restore hipblaslt and rocblas directories to torch/lib/ for ROCm-enabled PyTorch functionality
177COPY --from=torch-libs-chunker --chown=somebody /srv/torch_lib_chunks/hipblaslt/ "${TORCH_LIB_PATH}/hipblaslt/"
178COPY --from=torch-libs-chunker --chown=somebody /srv/torch_lib_chunks/rocblas/ "${TORCH_LIB_PATH}/rocblas/"
179
180# Copy pre-built wheels for MoRI, FlashAttention, aiter, and vLLM
181COPY --from=mori-builder --chown=somebody /srv/wheels /srv/wheels
182COPY --from=flashattention-builder --chown=somebody /srv/wheels /srv/wheels
183COPY --from=aiter-builder --chown=somebody /srv/wheels /srv/wheels
184COPY --from=vllm-builder --chown=somebody /srv/wheels /srv/wheels
185
186# Switch to user "somebody" to avoid running the container as root
187USER somebody
188
189# Install Python packages from wheels and uninstall unused package torchvision
190RUN pip install --no-cache-dir /srv/wheels/*.whl \
191 && pip uninstall -y torchvision \
192 # To reduce the image size, remove the wheel files as they are no longer needed
193 && rm -rf /srv/wheels/*
194
195# Set performance environment variables as defined in upstream vLLM ROCm dockerfiles
196ENV RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES=1 \
197 RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES=1 \
198 TOKENIZERS_PARALLELISM=false \
199 # ENV that can improve safe tensor loading, and end-to-end time
200 SAFETENSORS_FAST_GPU=1 \
201 # Performance environment variable.
202 HIP_FORCE_DEV_KERNARG=1 \
203 # Required for RCCL in ROCm7.1 (kept this because it also exists in the rocm/vllm:v0.14.0_amd_dev docker image that runs ROCm7.0)
204 HSA_NO_SCRATCH_RECLAIM=1

gerritbot added a comment.Feb 9 2026, 4:25 AM

Change #1237730 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/docker-images/production-images@master] ml: chunk torch libs in vLLM 0.14 image

https://gerrit.wikimedia.org/r/1237730

gerritbot added a project: Patch-For-Review.Feb 9 2026, 4:25 AM
gerritbot added a comment.Feb 9 2026, 8:48 AM

Change #1237730 merged by Dpogorzelski:

[operations/docker-images/production-images@master] ml: chunk torch libs in vLLM 0.14 image

https://gerrit.wikimedia.org/r/1237730

Maintenance_bot removed a project: Patch-For-Review.Feb 9 2026, 9:31 AM
kevinbazira added a comment.Feb 10 2026, 4:31 AM

The updated WMF Debian vLLM image that supports the latest upstream software stack as of Jan 2026 is now available in the wikimedia docker registry: https://docker-registry.wikimedia.org/ml/amd-vllm014/tags/

kevinbazira closed this task as Resolved.Feb 10 2026, 4:32 AM
kevinbazira moved this task from In Progress to 2025-2026 Q2 Done on the Machine-Learning-Team board.
kevinbazira mentioned this in T416966: Unable to pull vLLM images from Wikimedia docker registry due to authentication error.Feb 10 2026, 5:23 AM
gerritbot added a comment.Feb 10 2026, 5:47 AM

Change #1238235 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/docker-images/production-images@master] ml: correct minor version of vLLM 0.14 image

https://gerrit.wikimedia.org/r/1238235

gerritbot added a project: Patch-For-Review.Feb 10 2026, 5:47 AM
gerritbot added a comment.Feb 10 2026, 9:39 AM

Change #1238235 merged by Elukey:

[operations/docker-images/production-images@master] ml: correct minor version of vLLM 0.14 image

https://gerrit.wikimedia.org/r/1238235

Maintenance_bot removed a project: Patch-For-Review.Feb 10 2026, 10:31 AM
kevinbazira mentioned this in T418976: Migrate embeddings inference service from HF Transformers+CK FlashAttention to vLLM+AITER.Mar 4 2026, 8:09 AM
isarantopoulos moved this task from 2025-2026 Q2 Done to Task Archive on the Machine-Learning-Team board.May 20 2026, 11:13 AM
kevinbazira mentioned this in T428577: Update WMF Debian vLLM image to use pre-built wheels from upstream.Jun 9 2026, 11:12 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits