Page MenuHomePhabricator
Create Task
Maniphest T433977

Support OpenAI-compatible tool calling in the qwen36 model-server
Closed, ResolvedPublic
Actions

Description

Chat clients like the Open WebUI which we use in LiftWing Studio support "tool calling": the client sends the model a list of functions it may request (e.g a calculator, a clock, weather lookup, etc), the model replies with a structured request, the client runs the function and returns the result, and the model gives a final answer.

Our qwen36 model-server (behind the llm-qwen36-27b and llm-qwen3-14b services) does not support this: the KServe adapter it uses discards the tool list before it reaches the model. vLLM's built-in tool-calling flags only work in its own API server (see vLLM tool calling docs), which we do not use, so we shall implement support in our model-server.

In this task we shall:

  • Add tool-calling logic in model-server
  • Deploy model-server on LiftWing using TOOL_CALLING_ENABLED=True since it's False by default
  • Register a test tool in LiftWing Studio
  • Enable the test tool in a chat with the qwen model and use it

Acceptance criteria: Tool calling works for qwen models using the flow below:

  1. LiftWing Studio user sends What is the square root of 144?
  2. LiftWing model replies with <tool_call>{"name": "calculator", "arguments": {"equation": "sqrt(144)"}}</tool_call>
  3. LiftWing Studio runs calculator("sqrt(144)") locally, gets 12.0, appends the result to the conversation
  4. LiftWing model sees the result in the next turn and replies The square root of 144 is 12.0.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
ml-services: update qwen36-27b and qwen3-14b isvcs to support tool callingKevin Baziraoperations/deployment-chartsmaster+6 -2
qwen36: add tool calling, off by defaultKevin Baziramachinelearning/liftwing/inference-servicesmain+279 -9
Customize query in gerrit

Related Objects
Search...

Event Timeline

isarantopoulos created this task.Aug 4 2026, 1:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2026, 1:36 PM
isarantopoulos added a parent task: T433975: [draft] Q1 FY2026-27 LLM Platform Milestone 2.Aug 4 2026, 1:37 PM
kevinbazira renamed this task from [draft] explore vllm tool calling to Support OpenAI-compatible tool calling in the qwen36 model-server.Aug 5 2026, 5:11 AM
kevinbazira claimed this task.
kevinbazira updated the task description. (Show Details)
kevinbazira moved this task from Unsorted/Needs Triage to In Progress on the Machine-Learning-Team (Q1 FY2026-27) board.Aug 5 2026, 5:14 AM
gerritbot added a comment.Aug 5 2026, 5:28 AM

Change #1321272 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] qwen36: add tool calling, off by default

https://gerrit.wikimedia.org/r/1321272

gerritbot added a project: Patch-For-Review.Aug 5 2026, 5:28 AM
gerritbot added a comment.Aug 5 2026, 5:42 AM

Change #1321272 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] qwen36: add tool calling, off by default

https://gerrit.wikimedia.org/r/1321272

kevinbazira mentioned this in rMLIS5656d32b86b3: qwen36: add tool calling, off by default.Aug 5 2026, 5:43 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 5 2026, 6:31 AM
gerritbot added a comment.Aug 5 2026, 6:37 AM

Change #1321438 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: update qwen36-27b and qwen3-14b isvcs to support tool calling

https://gerrit.wikimedia.org/r/1321438

gerritbot added a project: Patch-For-Review.Aug 5 2026, 6:37 AM
gerritbot added a comment.Aug 5 2026, 6:43 AM

Change #1321438 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update qwen36-27b and qwen3-14b isvcs to support tool calling

https://gerrit.wikimedia.org/r/1321438

kevinbazira added a comment.Aug 5 2026, 7:22 AM

I have added tool calling in the qwen36 model-server and deployed it on LiftWing.

In LiftWing studio, I created the test tool as shown below:

Kevin Test Tool on LiftWing Studio.png (1,846×620 px, 115 KB)

P95907 Test tools used in LiftWing Studio for the Qwen model hosted using vLLM
1import os
2import requests
3from datetime import datetime
4from pydantic import BaseModel, Field
5
6class Tools:
7 def __init__(self):
8 pass
9
10 # Add your custom tools using pure Python code here, make sure to add type hints and descriptions
11
12 def get_user_name_and_email_and_id(self, __user__: dict = {}) -> str:
13 """
14 Get the user name, Email and ID from the user object.
15 """
16
17 # Do not include a descrption for __user__ as it should not be shown in the tool's specification
18 # The session user object will be passed as a parameter when the function is called
19
20 print(__user__)
21 result = ""
22
23 if "name" in __user__:
24 result += f"User: {__user__['name']}"
25 if "id" in __user__:
26 result += f" (ID: {__user__['id']})"
27 if "email" in __user__:
28 result += f" (Email: {__user__['email']})"
29
30 if result == "":
31 result = "User: Unknown"
32
33 return result
34
35 def get_current_time(self) -> str:
36 """
37 Get the current time in a more human-readable format.
38 """
39
40 now = datetime.now()
41 current_time = now.strftime("%I:%M:%S %p") # Using 12-hour format with AM/PM
42 current_date = now.strftime(
43 "%A, %B %d, %Y"
44 ) # Full weekday, month name, day, and year
45
46 return f"Current Date and Time = {current_date}, {current_time}"
47
48 def calculator(
49 self,
50 equation: str = Field(
51 ..., description="Math expression, e.g. sqrt(144)"
52 ),
53 ) -> str:
54 """Calculate the result of a math expression."""
55
56 # Avoid using eval in production code
57 # https://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html
58 import math
59 try:
60 return f"{equation} = {eval(equation, {'__builtins__': {}}, vars(math))}"
61 except Exception:
62 return "Invalid equation"
63
64 def get_current_weather(
65 self,
66 city: str = Field(
67 "New York, NY", description="Get the current weather for a given city."
68 ),
69 ) -> str:
70 """
71 Get the current weather for a given city.
72 """
73
74 api_key = os.getenv("OPENWEATHER_API_KEY")
75 if not api_key:
76 return (
77 "API key is not set in the environment variable 'OPENWEATHER_API_KEY'."
78 )
79
80 base_url = "http://api.openweathermap.org/data/2.5/weather"
81 params = {
82 "q": city,
83 "appid": api_key,
84 "units": "metric", # Optional: Use 'imperial' for Fahrenheit
85 }
86
87 try:
88 response = requests.get(base_url, params=params)
89 response.raise_for_status() # Raise HTTPError for bad responses (4xx and 5xx)
90 data = response.json()
91
92 if data.get("cod") != 200:
93 return f"Error fetching weather data: {data.get('message')}"
94
95 weather_description = data["weather"][0]["description"]
96 temperature = data["main"]["temp"]
97 humidity = data["main"]["humidity"]
98 wind_speed = data["wind"]["speed"]
99
100 return f"Weather in {city}: {temperature}°C"
101 except requests.RequestException as e:
102 return f"Error fetching weather data: {str(e)}"

After enabling the test tool in a LiftWing Studio chat with the Qwen3-14B model, I am able to use it as shown below:

Tool Calling on LiftWing Studio.png (1,850×1,056 px, 191 KB)

kevinbazira updated the task description. (Show Details)Aug 5 2026, 7:22 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 5 2026, 7:30 AM
kevinbazira added a comment.Aug 7 2026, 7:23 AM

Following yesterday's, ML knowledge sharing campfire meeting, I looked into whether we can create/update workspace tools without using the UI. The short answer: not declaratively yet. Open WebUI stores tools in its database, not as config files, so there is no native "drop a YAML/JSON in a directory and have it picked up" mechanism.

What does exist is a programmatic API:

  • POST /api/v1/tools/create : register a new tool
  • POST /api/v1/tools/id/{id}/update : update an existing tool
  • POST /api/v1/tools/id/{id}/access/update : set who can use it

The create endpoint accepts a JSON body with id, name, content (the Python source) and meta (description). Authentication is via Bearer token.

In practice for us, this means the closest thing to a "config-based deploy" would be an entrypoint script or CI/CD job that, when LiftWing Studio is healthy, curls the tool definitions to the API. Something like:

# In CI, after all checks have passed:
curl -s -X POST "https://liftwing-studio.wmcloud.org/api/v1/tools/create" \
    -H "Authorization: Bearer $LiftWing_Studio_API_KEY" \
    -H "Content-Type: application/json" \
    -d @tools/wikipedia_semantic_search.json

The tool source would still live in Gerrit or GitLab, get reviewed and merged, and the CI pipeline would push it to the API, with no manual UI interaction. It is an extra step compared to true declarative config, but it achieves the same outcome.

NOTE: the permission to create or update tools is effectively shell access on the server, because tool code runs as Python inside the Open WebUI container. So the API key should be tightly scoped and stored as a CI secret.
kevinbazira added a comment.EditedAug 7 2026, 7:54 AM

Here is a tool calling demo for Wikipedia search suggested by Ilias:

The tool's code is shown below:

P95917 Wikipedia Semantic Search tool used in LiftWing Studio for the Qwen model hosted using vLLM
1"""
2title: Wikipedia Semantic Search
3description: Search Wikipedia articles by meaning, not just keywords, in any language edition.
4"""
5
6import re
7
8import requests
9
10USER_AGENT = "liftwing-studio-tools/0.1 (ml-team; ml@wikimedia.org)"
11TIMEOUT = 10
12EXTRACT_MAX_CHARS = 500
13DEFAULT_LANGUAGE = "en"
14DEFAULT_LIMIT = 5
15MAX_LIMIT = 10
16
17
18class Tools:
19 def __init__(self):
20 # Ask Open WebUI to render this tool's output as a citation in the UI,
21 # independently of whether the model chooses to cite it in its reply.
22 # Support varies by Open WebUI version; harmless where unsupported.
23 self.citation = True
24
25 def wikipedia_semantic_search(
26 self,
27 query: str,
28 language: str = DEFAULT_LANGUAGE,
29 limit: int = DEFAULT_LIMIT,
30 ) -> str:
31 """
32 Search Wikipedia for articles matching a natural-language query.
33 Uses semantic (meaning-based) search where available, falling back
34 to keyword search. Returns titles, descriptions, extracts, and URLs.
35 :param query: Natural-language search query
36 :param language: Wikipedia language code, e.g. 'en', 'de', 'sw'
37 :param limit: Maximum number of results (1-10)
38 """
39 # Defensive coercion. Open WebUI calls this as a plain function, so
40 # any argument the model omits arrives as the declared default and
41 # any argument it supplies arrives unvalidated. Do not use pydantic
42 # Field() defaults here: an omitted argument would then be a
43 # FieldInfo object rather than a value, and int(FieldInfo) raises
44 # TypeError before the request is ever made.
45 if not isinstance(query, str) or not query.strip():
46 return "Tool input error, do not retry: no search query provided."
47 query = query.strip()
48 if not isinstance(language, str) or not language.replace("-", "").isalnum():
49 language = DEFAULT_LANGUAGE
50 try:
51 limit = max(1, min(int(limit), MAX_LIMIT))
52 except (TypeError, ValueError):
53 limit = DEFAULT_LIMIT
54
55 headers = {"User-Agent": USER_AGENT, "Accept": "application/json"}
56 base = f"https://{language}.wikipedia.org"
57
58 # 1. Action API list=search with the experimental semantic parameter.
59 # The action API's parameter validator does not declare
60 # cirrusSemanticSearch and so returns an "Unrecognized parameter"
61 # warning, but the search backend still honours it: verified on
62 # en.wikipedia.org on 2026-08-05 by comparing result sets with and
63 # without the parameter (deterministic and clearly different).
64 # On wikis where it has no effect, results degrade to keyword
65 # ranking, so callers always get a result list.
66 # searchinfo.totalhits is deliberately not surfaced: under
67 # semantic search it tracks the candidate-set size (it scales with
68 # srlimit) rather than a real match count, so reporting it would
69 # invite the model to state a fabricated total.
70 params = {
71 "action": "query",
72 "format": "json",
73 "formatversion": "2",
74 "list": "search",
75 "srsearch": query,
76 "srprop": "snippet|sectiontitle",
77 "srlimit": str(limit),
78 "srnamespace": "0",
79 "cirrusSemanticSearch": "",
80 }
81 try:
82 r = requests.get(
83 f"{base}/w/api.php", params=params, headers=headers, timeout=TIMEOUT
84 )
85 r.raise_for_status()
86 hits = r.json().get("query", {}).get("search", [])
87 except requests.RequestException as e:
88 return f"Wikipedia search request failed, do not retry: {e}"
89 except ValueError as e:
90 return f"Wikipedia search returned an unreadable response, do not retry: {e}"
91
92 if not hits:
93 return (
94 f"No Wikipedia articles found for '{query}' on "
95 f"{language}.wikipedia.org."
96 )
97
98 # 2. Per-hit enrichment via the REST page-summary endpoint
99 # (extract, short description, canonical URL). A failed summary
100 # call degrades gracefully to the search snippet.
101 lines = []
102 for i, hit in enumerate(hits, 1):
103 title = hit.get("title", "")
104 if not title:
105 continue
106 snippet = _strip_html(hit.get("snippet", ""))
107 section = hit.get("sectiontitle")
108 extract = snippet
109 description = None
110 url = f"{base}/wiki/{requests.utils.quote(title.replace(' ', '_'))}"
111
112 try:
113 s = requests.get(
114 f"{base}/api/rest_v1/page/summary/"
115 f"{requests.utils.quote(title, safe='')}",
116 headers=headers,
117 timeout=TIMEOUT,
118 )
119 if s.ok:
120 data = s.json()
121 extract = data.get("extract") or snippet
122 description = data.get("description")
123 url = (
124 data.get("content_urls", {}).get("desktop", {}).get("page")
125 or url
126 )
127 except (requests.RequestException, ValueError):
128 pass
129
130 if len(extract) > EXTRACT_MAX_CHARS:
131 extract = extract[:EXTRACT_MAX_CHARS].rsplit(" ", 1)[0] + "..."
132
133 desc = f" ({description})" if description else ""
134 sect = f" [section: {section}]" if section else ""
135 lines.append(f"{i}. {title}{desc}{sect}\n {extract}\n Source: {url}")
136
137 if not lines:
138 return (
139 f"No usable Wikipedia results for '{query}' on "
140 f"{language}.wikipedia.org."
141 )
142
143 # In-band instruction. Asking for citations in the tool output rather
144 # than relying on the user's phrasing makes the behaviour hold on every
145 # call. The second clause targets residual overclaiming: on 2026-08-05
146 # a 14B model summarised unrelated articles as though they answered the
147 # query, and requesting citations reduced but did not remove this.
148 return (
149 f"Wikipedia results for '{query}' ({language}). When answering, cite "
150 "the Source URL for each article you reference, and do not claim an "
151 "article covers a topic unless its extract says so:\n\n"
152 + "\n\n".join(lines)
153 )
154
155
156def _strip_html(html: str) -> str:
157 """Strip the lightweight HTML the action API includes in snippets."""
158 if not isinstance(html, str):
159 return ""
160 text = re.sub(r"<[^>]+>", "", html)
161 for a, b in [
162 ("&nbsp;", " "),
163 ("&amp;", "&"),
164 ("&lt;", "<"),
165 ("&gt;", ">"),
166 ("&quot;", '"'),
167 ("&#39;", "'"),
168 ]:
169 text = text.replace(a, b)
170 return " ".join(text.split())

kevinbazira closed this task as Resolved.Aug 7 2026, 8:14 AM

Closing this task as we've managed to enable tool calling in a LiftWing model-server, create tools in LiftWing Studio, test them in chat, and get grounded responses.

As discussed in the ML team meeting, we will enable additional tools as we refine the WMF LLM platform personas and their priority use-cases. That work will be tracked in a separate task.

kevinbazira moved this task from In Progress to Done on the Machine-Learning-Team (Q1 FY2026-27) board.Aug 7 2026, 8:14 AM
isarantopoulos mentioned this in T434274: Explore LLM tool calling support in our vLLM-based inference services.Aug 7 2026, 9:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits