Summary
This task is about LLM tool calling (function calling) at the model-server level: the ability of a Lift Wing inference service to accept OpenAI-style tools definitions in the request and return parsed tool_calls in the response, as supported by vLLM. This is server-side functionality of the LLM endpoint itself, not to be confused with enabling tools in a client application or UI (e.g. Open WebUI), which merely consumes this capability. A test showed tool calling already works on the llm-qwen3-14b endpoint. in T433977: Support OpenAI-compatible tool calling in the qwen36 model-server. We want to understand how it is enabled in vLLM, whether it works reliably across the LLMs we host, and whether we can offer it as a supported feature.
Technical notes
In vLLM, tool call parsing happens in the OpenAI-compatible serving layer (not the engine) and is enabled with --enable-auto-tool-choice plus a model-family-specific --tool-call-parser (e.g. hermes for Qwen models, llama3_json for Llama 3.x).
example request
curl 'https://api.wikimedia.org/service/lw/inference/v1/models/llm-qwen3-14b/openai/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'User-Agent: my-tool/0.1 (you@wikimedia.org)' \
-d '{
"model": "llm-qwen3-14b",
"messages": [{"role": "user", "content": "A user question the tool can help answer"}],
"tools": [{
"type": "function",
"function": {
"name": "my_tool_name",
"description": "Description of what the tool does",
"parameters": {
"type": "object",
"properties": {"my_param": {"type": "string", "description": "Description of the parameter"}},
"required": ["my_param"]
}
}
}],
"tool_choice": "auto",
"temperature": 0
}'The response contains a parsed tool call instead of text content:
"message": {"role": "assistant", "content": null,
"tool_calls": [{"id": "call_xxxxxxxx", "type": "function",
"function": {"name": "my_tool_name", "arguments": "{\"my_param\": \"value chosen by the model\"}"}}]},
"finish_reason": "tool_calls"Links:
- vLLM tool calling docs: https://docs.vllm.ai/en/latest/features/tool_calling.html
- Chat completions endpoint function/tool calling - https://developers.openai.com/api/docs/guides/function-calling
Acceptance criteria
- We understand how tool calling is enabled in vLLM (e.g. tool call parser configuration per model family) and which Lift Wing LLMs support it.
- We deploy and test a specific tool on the server side and to the documentation how such requests can be made. (currently test tools are hosted, we'll run load tests on tools that will receive real traffic)
- We have a recommendation on whether to officially support and document tool calling on Lift Wing. (see T434274#12256487)
