Regarding the open questions raised above:

I've also created small local development environment, which is here: https://gitlab.wikimedia.org/repos/machine-learning/liftwing-studio/-/tree/bartosz-librechat/experiments/bartosz-librechat?ref_type=heads.

Just to confirm: You would like to import https://github.com/mediawiki-utilities/python-mwapi right?

We discussed the options and want to go with forking mwapi and maintaining it ourselves, hosted on Wikimedia GitLab. Main reasons are close-to-zero migration cost + it unblocks T363725, T407843 and lets us land fixes to mwapi. GitLab seems preferred as it's within our own infrastructure with our own CI, compared to GitHub.
@Dzahn Would you be able to help us create the repo and import the code? Does repos/machine-learning/python-mwapi sound like a good path, or do you prefer a different namespace?

4. Proposal of the path forward

3. What are the current problems we face with LLM serving on LiftWing

2. How do we currently serve LLMs on LiftWing

I've done some reading, thinking and designing. I will structure the incoming comments into a few main sections:

We've just enabled the new languages (es, ru, ja, nl, zh, de) for the Revise Tone pipeline.
Service logs confirm we're successfully processing edits from all new wikis. We will keep monitoring service metrics, but everything looks healthy so far. I am resolving the ticket :)

Are these just limitations of using the topics model, with its existing imperfections? Or something that is happening due to filtering we added for the corpus. I ask because I recognize the wrestling-bias, we saw many wrestlers when we used pageviews as an index for the explore feed topic interest suggestions. I am guessing that would improve if we removed this qualifier: 70th–90th percentile of pageviews for the topic.

Thank you! I've implemented the changes: keyword filtering applied to all articles in the list and I'm using more recommended articles in cases where we can't fill the full seed list for a topic. I'm attaching a resulting dataset here:

I investigated a little option 1 of switching to mwclient library for inference-services, but I don't think it covers our use-cases well, mostly the async support needed for async KServe services.

Nice! I think for LAC uses that need to be triggered by edits, we can get you all off of change prop entirely. The business of producing events via eventgate will move into MediaWiki (probably in WikimediaEvents extension?).

FYI we have added ephemeral-storage requests/limits to all predictor containers that we are serving on the LiftWing cluster and everything is now redeployed with storage declared on staging and prod clusters. It was tracked here: https://phabricator.wikimedia.org/T431089.

We have now disabled the topic filtering alltogether.

This task should be picked up only after we resolve the work tracked in here https://phabricator.wikimedia.org/T434763 since it relies heavily on the underlying package we use for MW API calls.

The second version of the dataset is now created and available here:

We've done a first bump of allowed topics to 30 out of 64 topics covering Culture, History_and_Society topics plus STEM.Technology and there are no problems so far.
We'll let it run for a day to make sure everything is working smoothly and we'll enable remaining topics if there will be no problems.

I've also updated non-KServe deployments to include ephemeral-storage limits/requests and re-deployed them across staging and prod clusters.

Thank you for the review!

All KServe deployments on staging and production are now re-deployed with ephemeral-storage request and limits declared. During the work on this task I also noticed that we have some deployments not declaring memory and cpu requests/limits either, which can be tackled in a follow up ticket. Once this will be done, we could add requirements for all cpu/memory/storage be required on the chart level.

The gpt-oss-safeguard-20b service was hardened such that if the model output cannot be parsed as Harmony messages, was truncated at max_tokens, or contains no final channel (degenerate generation on pathological input, or reasoning loops exhausting the token budget), the server returns a readable error response instead of an HTTP 500 or a bogus verdict.

I've performed first full run of dataset generation and I'm attaching the resulting dataset csv file here (columns: Topic, Article title, Article URL, Type, Edit rate in the last 7 days).

In T429540#12159144, @Ottomata wrote:

@BWojtowicz-WMF , should this be using the new Linked Artifacts Cache topics endpoint?

I've started working on the script to generate the dataset and I've come across a few questions / things to be confirmed:

Closing this for now. The underlying issue of corrupted responses on distributed inference is not fully solved, but we agreed to not do distributed inference on partitioned MI300X GPUs in line with AMD docs.

The idea of warming up LAC this way via MW JobQueue sounds great to me!

We can re-use this task to decide if we want to do backfilling and how to perform it if yes :)

The model is now deployed in the llm namespace and is exposed via REST Gateway with LiftWingLLM rate limits. Additionally, I've added it to LiftWing studio so everyone can play around with it via UI.
Model can be queried like below:

curl -s https://api.wikimedia.org/service/lw/inference/v1/models/llm-qwen36-27b/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llm-qwen36-27b","messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":200}' | jq .

I removed the DTYPE=float16 and exchanged it with DTYPE=auto env var and service still responds happily, which confirms that the FP8 quantization works.

I've done first batch of updates targeting our LLM models, each of them now is now defining both requests and limits for ephemeral storage. All models are redeployed.

Regarding timeouts, our service definitely takes _a lot_ of time to respond (with success!), which results in the 10 second timeout:

INFO:root:Opening a new Asyncio session for restgateway.
2026-07-08 15:34:46.402 kserve.trace requestId: a6e43b78-520a-4e53-9e6d-a2f428de93f3, preprocess_ms: 311.614513397, explain_ms: 0, predict_ms: 11384.052991867, postprocess_ms: 0.011920929
2026-07-08 15:34:46.402 uvicorn.access INFO:     127.0.0.6:0 1 - "POST /v1/models/article-descriptions%3Apredict HTTP/1.1" 200 OK
2026-07-08 15:34:46.403 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 11.697105884552002
2026-07-08 15:34:46.403 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 22.731085999999777
2026-07-08 15:34:49.880 uvicorn.access INFO:     127.0.0.6:43489 1 - "GET /metrics HTTP/1.1" 200 OK
2026-07-08 15:34:49.880 kserve.trace kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0014925003051757812
2026-07-08 15:34:49.881 kserve.trace kserve.io.kserve.protocol.rest.server.metrics_handler: 0.001480999999330379
2026-07-08 15:35:02.738 uvicorn.access INFO:     127.0.0.6:60479 1 - "GET /metrics HTTP/1.1" 200 OK
2026-07-08 15:35:02.738 kserve.trace kserve.io.kserve.protocol.rest.server.metrics_handler: 0.001093149185180664
2026-07-08 15:35:02.738 kserve.trace kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0010870000005525071
INFO:root:Opening a new Asyncio session for restgateway.
2026-07-08 15:35:27.365 kserve.trace requestId: b0ae7ad0-1133-4d81-8e1c-a6f65d322cd3, preprocess_ms: 242.444753647, explain_ms: 0, predict_ms: 11316.508769989, postprocess_ms: 0.016212463

We've noticed similar issue with corrupt responses when deploying qwen3-14b model on 2x 24GB VRAM partitions of MI300X - https://phabricator.wikimedia.org/T431268.

Thank you for your inputs @achou @isarantopoulos! It seems we are all agreeing on Option 1, which means tainting all current GPU nodes.

Moved the experimental qwen3-14b deployment to a single 192GB GPU on ml-serve1012 and everything works fine. Same change for llm-qwen3-14b is on the way.

One potential issue I've noticed in startup logs is this:

(EngineCore_DP0 pid=186) [aiter] WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'" for more details: https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html#disable-numa-auto-balancing
(EngineCore_DP0 pid=186) [2026-07-06 09:02:14] WARNING core.py:454: WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'" for more details: https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html#disable-numa-auto-balancing

The changes to the gateway (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1305621) are merged and deployed. We've settled on this rate limiting policy:

"LiftWingLLM":
  shadow_mode: false
  limits:
    "*": # strict shared limit for public access (anon*, unauthed*, authed-*)
      HOUR: 100
    "known-network": # network under our control: WMCS / Toolforge (x-trusted-request: A)
      HOUR: 9999999
    "known-client": # network associated with a known client (x-trusted-request: B)
      HOUR: 9999999
    "approved-bot": # community approved bot, based on jwt auth
      HOUR: 9999999

I've drafted the REST Gateway changes in here https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1305621.
I went for the llm-* path matching routing to llm namespace. So all services matching the llm-* regex inside llm namespace would be automatically exposed with the LiftWingLLM rate limiting policy. I included conservative 100 requests per hour limit for public traffic according to the discussion above.

I've looked into the current rate limiting setup for LiftWing and have questions and ideas on how we could approach it. We'd love to push this work forwards so I'd love to hear inputs from ServiceOps and MW-Interfaces.
Pinging @daniel and @Clement_Goubert I got a suggestion to include you here :)

First, let me give some context on options I'll share below - taints/tolerations and nodeAffinity do opposite jobs: a taint repels everything that does not explicitly tolerate it, while nodeAffinity only attracts a pod when it is present, but does not do any "blocking" during scheduling. The practical consequence is that if we want hard guarantees about which GPUs we land on, it has to come from taints as affinities cannot prevent accidental schedules.

Current state.

Update
Two patches landed to improve our MW API call queries within Article Topics

• First change, which was related to passing invalid revision_id usually linked to deleted/renamed/moved pages. This previously was throwing 500 error without explanation, but now throws a nice 400 with explanation.
• Second change added retries on MW API calls, which failed for transient reasons. We are doing a lot of MW API calls for single request and this should allow us to have lower error rate for Article Topic requests.

Status Update

We've turned on the integration on production as well 🎉
Example of how to query Hoarde is below, it is only reachable internally at the moment so you need to be e.g. on a stat host.

# Production:
curl -D - https://linked-artifacts.discovery.wmnet:30443/revisions/v1/article_topics/enwiki/39755715/1235690033; echo
# Staging:
curl -D - https://linked-artifacts.k8s-staging.discovery.wmnet:30443/revisions/v1/article_topics/enwiki/39755715/1235690033; echo

I did some more digging on the 500s.
Big chunk of those errors stems from the fact that I was using ~year old revision_id, which later broke our MW API calls (using action=parse&oldid=... call), because some of those pages have been deleted/suppressed/renamed. Such calls make MW API return an error ( nusuchrevid / missingtitle ), which we currently surface as generic 500.
Good news is that in practice we would be usually using fresh revision_ids (e.g. by page_change events), which would mean that error rate would be lower. However, using old revision_ids is still a legitimate use-case, which we should be able to handle.

Status update

Status Update

We confirmed that gRPC endpoints works via standard 30443 port on production server without the LVS changes. Glad that we found that out, thank you @JMeybohm and @elukey!

@elukey
Thanks, I indeed missed it! Initially I thought that 2nd gateway might be a way to overcome the Knatives auto-managed Gateway single-port limitation, but now I see that it is not the case.
I indeed just tested the script above, but pointing at the 30443 port directly instead of the new 30051 port and it succeeded, which proves that the LVS changes were indeed not needed(?).

Thanks to the changes to LVS, I was successful with testing the gRPC connection on staging with the script below! 🎉
We will now do the gRPC integration with Hoarde on staging and if everything goes smooth, we can merge similar LVS changes to production. Thank you for all the work here!

I'm sharing a draft of solution architecture we discussed in our ML Team Meeting for this problem. The approach is similar to the prototype solution developed by Kevin.
It could accommodate both batch jobs and real-time updates. End users would communicate with the backend via /generate endpoint to enqueue new TTS jobs or via /audio backend to retrieve path to the generated TTS files.

Happy update!

Update on current state of things.