Errors on llm-qwen3-14b deployment after moving it to ml-serve1015 24GB partition size
ActivePublic
Actions

Authored by gkyziridis on Fri, Oct 2, 11:21 AM.

Project Tags

None

Referenced Files

	F105836569: Errors on llm-qwen3-14b deployment after moving it to ml-serve1015 24GB partition size
	Fri, Oct 2, 11:21 AM

Subscribers

None

	$ kubectl -n llm describe isvc llm-qwen3-14b
	File "/srv/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 246, in from_engine_args
	return cls(
	^^^^
	File "/srv/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
	self.engine_core = EngineCoreClient.make_async_mp_client(
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/srv/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
	return func(args, *kwargs)
	^^^^^^^^^^^^^^^^^^^^^
	File "/srv/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 131, in make_async_mp_client
	return AsyncMPClient(*client_args)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/srv/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
	return func(args, *kwargs)
	^^^^^^^^^^^^^^^^^^^^^
	File "/srv/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 932, in __init__
	super().__init__(
	File "/srv/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 567, in __init__
	with launch_core_engines(
	^^^^^^^^^^^^^^^^^^^^
	File "/opt/python3.12/lib/python3.12/contextlib.py", line 144, in __exit__
	next(self.gen)
	File "/srv/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1150, in launch_core_engines
	wait_for_engine_startup(
	File "/srv/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1209, in wait_for_engine_startup
	raise RuntimeError(
	RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/srv/app/model.py", line 436, in <module>
	model.load()
	File "/srv/app/model.py", line 115, in load
	raise kserve.errors.ModelMissingError(error_message)
	kserve.errors.ModelMissingError: Failed to load model. Reason: Engine core initialization failed. See root cause above. Failed core proc(s): {}

	-------

	$ POD=llm-qwen3-14b-predictor-00011-deployment-79dffcd5f4-cgttx
	$ kubectl -n llm logs $POD -c kserve-container \| grep -Ei "KV cache\|concurrency\|memory\|error"
	# This error was printed 20 times just copy paste one.

	ERROR:root:Error during inference: This model's maximum context length is 16384 tokens. However, you requested 0 output tokens and your prompt contains at least 16385 input tokens, for a total of at least 16385 tokens. Please reduce the length of the input prompt or the number of requested output tokens. (parameter=input_tokens, value=16385)
	ERROR:kserve:Exception:
	raise VLLMValidationError(
	vllm.exceptions.VLLMValidationError: This model's maximum context length is 16384 tokens. However, you requested 0 output tokens and your prompt contains at least 16385 input tokens, for a total of at least 16385 tokens. Please reduce the length of the input prompt or the number of requested output tokens. (parameter=input_tokens, value=16385)
	raise InferenceError(error_message)
	kserve.errors.InferenceError: Error during inference: This model's maximum context length is 16384 tokens. However, you requested 0 output tokens and your prompt contains at least 16385 input tokens, for a total of at least 16385 tokens. Please reduce the length of the input prompt or the number of requested output tokens. (parameter=input_tokens, value=16385)

Event Timeline

gkyziridis created this paste.Fri, Oct 2, 11:21 AM

gkyziridis mentioned this in T439555: Move llm isvcs to new partitions.Fri, Oct 2, 12:18 PM

Errors on llm-qwen3-14b deployment after moving it to ml-serve1015 24GB partition sizeActivePublicActions

Event Timeline

Errors on llm-qwen3-14b deployment after moving it to ml-serve1015 24GB partition size
ActivePublic
Actions