Page MenuHomePhabricator

Qwen3-14b responses are corrupted when running on 2x24GB partitions of MI300x
Closed, ResolvedPublicBUG REPORT

Description

Qwen3-14b was successfully running on 2x192GB partitions on ml-serve1012 host.

We tried moving the host to ml-serve1013 to use 2x24GB partitions to host the model: patch1, patch2.

The deployment was successful and the model server started. However, the output from the model was corrupted - it returned the same token multiple times, which suggests there are numerical present in the model.

Example:

~ curl -s --http1.1 "https://inference.svc.eqiad.wmnet:30443/openai/v1/chat/completions"   -H "Host: llm-qwen3-14b-predictor.llm.wikimedia.org"   -H "Content-Type: application/json"   -d '{"model":"llm-qwen3-14b","messages":[{"role":"user","content":"What is 2+2?"}],"max_tokens":50}'

{"id":"990516f0c6255553","object":"chat.completion","created":1783332682,"model":"llm-qwen3-14b","choices":[{"index":0,"message":{"role":"assistant","content":"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null,"reasoning_content":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":19,"total_tokens":69,"completion_tokens":50,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}

Event Timeline

This seems related to the qwen3.6-27b deployment attempt on multiple partitions in https://phabricator.wikimedia.org/T425680#12059487

Change #1307749 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: run experimental qwen3-14b on a single GPU on ml-serve1012

https://gerrit.wikimedia.org/r/1307749

One potential issue I've noticed in startup logs is this:

(EngineCore_DP0 pid=186) [aiter] WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'" for more details: https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html#disable-numa-auto-balancing
(EngineCore_DP0 pid=186) [2026-07-06 09:02:14] WARNING core.py:454: WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'" for more details: https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html#disable-numa-auto-balancing

Change #1307749 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: run experimental qwen3-14b on a single GPU on ml-serve1012

https://gerrit.wikimedia.org/r/1307749

Change #1307757 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: run llm-qwen3-14b on a single GPU on ml-serve1012

https://gerrit.wikimedia.org/r/1307757

Moved the experimental qwen3-14b deployment to a single 192GB GPU on ml-serve1012 and everything works fine. Same change for llm-qwen3-14b is on the way.

One note useful in relation to https://phabricator.wikimedia.org/T425680#12059487:
We see the same signature of !!!! in responses and we've noted one potential issue to be missing FP8 kernel configs. However, qwen3-14b doesn't use the fp8 path and we have the same issue so that can't be the only explanation. It seems to common factor is the partitioning schema.

Change #1307757 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: run llm-qwen3-14b on a single GPU on ml-serve1012

https://gerrit.wikimedia.org/r/1307757

Change #1308036 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Update qwen3-14b MODEL_NAME env var.

https://gerrit.wikimedia.org/r/1308036

Change #1308085 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Remove experimental qwen3-14b deployment.

https://gerrit.wikimedia.org/r/1308085

Change #1308085 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Remove experimental qwen3-14b deployment.

https://gerrit.wikimedia.org/r/1308085

Change #1308582 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: run qwen36-27b on a single GPU on ml-serve1012

https://gerrit.wikimedia.org/r/1308582

Change #1308582 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: run qwen36-27b on a single GPU on ml-serve1012

https://gerrit.wikimedia.org/r/1308582

Closing this for now. The underlying issue of corrupted responses on distributed inference is not fully solved, but we agreed to not do distributed inference on partitioned MI300X GPUs in line with AMD docs.

Change #1308036 abandoned by Bartosz Wójtowicz:

[operations/deployment-charts@master] ml-services: Update qwen3-14b MODEL_NAME env var.

https://gerrit.wikimedia.org/r/1308036