FYI we have added ephemeral-storage requests/limits to all predictor containers that we are serving on the LiftWing cluster and everything is now redeployed with storage declared on staging and prod clusters. It was tracked here: https://phabricator.wikimedia.org/T431089.

We have now disabled the topic filtering alltogether.

This task should be picked up only after we resolve the work tracked in here https://phabricator.wikimedia.org/T434763 since it relies heavily on the underlying package we use for MW API calls.

The second version of the dataset is now created and available here:

We've done a first bump of allowed topics to 30 out of 64 topics covering Culture, History_and_Society topics plus STEM.Technology and there are no problems so far.
We'll let it run for a day to make sure everything is working smoothly and we'll enable remaining topics if there will be no problems.

I've also updated non-KServe deployments to include ephemeral-storage limits/requests and re-deployed them across staging and prod clusters.

Thank you for the review!

All KServe deployments on staging and production are now re-deployed with ephemeral-storage request and limits declared. During the work on this task I also noticed that we have some deployments not declaring memory and cpu requests/limits either, which can be tackled in a follow up ticket. Once this will be done, we could add requirements for all cpu/memory/storage be required on the chart level.

The gpt-oss-safeguard-20b service was hardened such that if the model output cannot be parsed as Harmony messages, was truncated at max_tokens, or contains no final channel (degenerate generation on pathological input, or reasoning loops exhausting the token budget), the server returns a readable error response instead of an HTTP 500 or a bogus verdict.

I've performed first full run of dataset generation and I'm attaching the resulting dataset csv file here (columns: Topic, Article title, Article URL, Type, Edit rate in the last 7 days).

In T429540#12159144, @Ottomata wrote:

@BWojtowicz-WMF , should this be using the new Linked Artifacts Cache topics endpoint?

I've started working on the script to generate the dataset and I've come across a few questions / things to be confirmed:

Closing this for now. The underlying issue of corrupted responses on distributed inference is not fully solved, but we agreed to not do distributed inference on partitioned MI300X GPUs in line with AMD docs.

The idea of warming up LAC this way via MW JobQueue sounds great to me!

We can re-use this task to decide if we want to do backfilling and how to perform it if yes :)

The model is now deployed in the llm namespace and is exposed via REST Gateway with LiftWingLLM rate limits. Additionally, I've added it to LiftWing studio so everyone can play around with it via UI.
Model can be queried like below:

curl -s https://api.wikimedia.org/service/lw/inference/v1/models/llm-qwen36-27b/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llm-qwen36-27b","messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":200}' | jq .

I removed the DTYPE=float16 and exchanged it with DTYPE=auto env var and service still responds happily, which confirms that the FP8 quantization works.

I've done first batch of updates targeting our LLM models, each of them now is now defining both requests and limits for ephemeral storage. All models are redeployed.

Regarding timeouts, our service definitely takes _a lot_ of time to respond (with success!), which results in the 10 second timeout:

INFO:root:Opening a new Asyncio session for restgateway.
2026-07-08 15:34:46.402 kserve.trace requestId: a6e43b78-520a-4e53-9e6d-a2f428de93f3, preprocess_ms: 311.614513397, explain_ms: 0, predict_ms: 11384.052991867, postprocess_ms: 0.011920929
2026-07-08 15:34:46.402 uvicorn.access INFO:     127.0.0.6:0 1 - "POST /v1/models/article-descriptions%3Apredict HTTP/1.1" 200 OK
2026-07-08 15:34:46.403 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 11.697105884552002
2026-07-08 15:34:46.403 kserve.trace kserve.io.kserve.protocol.rest.v1_endpoints.predict: 22.731085999999777
2026-07-08 15:34:49.880 uvicorn.access INFO:     127.0.0.6:43489 1 - "GET /metrics HTTP/1.1" 200 OK
2026-07-08 15:34:49.880 kserve.trace kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0014925003051757812
2026-07-08 15:34:49.881 kserve.trace kserve.io.kserve.protocol.rest.server.metrics_handler: 0.001480999999330379
2026-07-08 15:35:02.738 uvicorn.access INFO:     127.0.0.6:60479 1 - "GET /metrics HTTP/1.1" 200 OK
2026-07-08 15:35:02.738 kserve.trace kserve.io.kserve.protocol.rest.server.metrics_handler: 0.001093149185180664
2026-07-08 15:35:02.738 kserve.trace kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0010870000005525071
INFO:root:Opening a new Asyncio session for restgateway.
2026-07-08 15:35:27.365 kserve.trace requestId: b0ae7ad0-1133-4d81-8e1c-a6f65d322cd3, preprocess_ms: 242.444753647, explain_ms: 0, predict_ms: 11316.508769989, postprocess_ms: 0.016212463

We've noticed similar issue with corrupt responses when deploying qwen3-14b model on 2x 24GB VRAM partitions of MI300X - https://phabricator.wikimedia.org/T431268.

Thank you for your inputs @achou @isarantopoulos! It seems we are all agreeing on Option 1, which means tainting all current GPU nodes.

Moved the experimental qwen3-14b deployment to a single 192GB GPU on ml-serve1012 and everything works fine. Same change for llm-qwen3-14b is on the way.

One potential issue I've noticed in startup logs is this:

(EngineCore_DP0 pid=186) [aiter] WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'" for more details: https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html#disable-numa-auto-balancing
(EngineCore_DP0 pid=186) [2026-07-06 09:02:14] WARNING core.py:454: WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'" for more details: https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html#disable-numa-auto-balancing

The changes to the gateway (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1305621) are merged and deployed. We've settled on this rate limiting policy:

"LiftWingLLM":
  shadow_mode: false
  limits:
    "*": # strict shared limit for public access (anon*, unauthed*, authed-*)
      HOUR: 100
    "known-network": # network under our control: WMCS / Toolforge (x-trusted-request: A)
      HOUR: 9999999
    "known-client": # network associated with a known client (x-trusted-request: B)
      HOUR: 9999999
    "approved-bot": # community approved bot, based on jwt auth
      HOUR: 9999999

I've drafted the REST Gateway changes in here https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1305621.
I went for the llm-* path matching routing to llm namespace. So all services matching the llm-* regex inside llm namespace would be automatically exposed with the LiftWingLLM rate limiting policy. I included conservative 100 requests per hour limit for public traffic according to the discussion above.

I've looked into the current rate limiting setup for LiftWing and have questions and ideas on how we could approach it. We'd love to push this work forwards so I'd love to hear inputs from ServiceOps and MW-Interfaces.
Pinging @daniel and @Clement_Goubert I got a suggestion to include you here :)

First, let me give some context on options I'll share below - taints/tolerations and nodeAffinity do opposite jobs: a taint repels everything that does not explicitly tolerate it, while nodeAffinity only attracts a pod when it is present, but does not do any "blocking" during scheduling. The practical consequence is that if we want hard guarantees about which GPUs we land on, it has to come from taints as affinities cannot prevent accidental schedules.

Current state.

Update
Two patches landed to improve our MW API call queries within Article Topics

• First change, which was related to passing invalid revision_id usually linked to deleted/renamed/moved pages. This previously was throwing 500 error without explanation, but now throws a nice 400 with explanation.
• Second change added retries on MW API calls, which failed for transient reasons. We are doing a lot of MW API calls for single request and this should allow us to have lower error rate for Article Topic requests.

Status Update

We've turned on the integration on production as well 🎉
Example of how to query Hoarde is below, it is only reachable internally at the moment so you need to be e.g. on a stat host.

# Production:
curl -D - https://linked-artifacts.discovery.wmnet:30443/revisions/v1/article_topics/enwiki/39755715/1235690033; echo
# Staging:
curl -D - https://linked-artifacts.k8s-staging.discovery.wmnet:30443/revisions/v1/article_topics/enwiki/39755715/1235690033; echo

I did some more digging on the 500s.
Big chunk of those errors stems from the fact that I was using ~year old revision_id, which later broke our MW API calls (using action=parse&oldid=... call), because some of those pages have been deleted/suppressed/renamed. Such calls make MW API return an error ( nusuchrevid / missingtitle ), which we currently surface as generic 500.
Good news is that in practice we would be usually using fresh revision_ids (e.g. by page_change events), which would mean that error rate would be lower. However, using old revision_ids is still a legitimate use-case, which we should be able to handle.

Status update

Status Update

We confirmed that gRPC endpoints works via standard 30443 port on production server without the LVS changes. Glad that we found that out, thank you @JMeybohm and @elukey!

@elukey
Thanks, I indeed missed it! Initially I thought that 2nd gateway might be a way to overcome the Knatives auto-managed Gateway single-port limitation, but now I see that it is not the case.
I indeed just tested the script above, but pointing at the 30443 port directly instead of the new 30051 port and it succeeded, which proves that the LVS changes were indeed not needed(?).

Thanks to the changes to LVS, I was successful with testing the gRPC connection on staging with the script below! 🎉
We will now do the gRPC integration with Hoarde on staging and if everything goes smooth, we can merge similar LVS changes to production. Thank you for all the work here!

I'm sharing a draft of solution architecture we discussed in our ML Team Meeting for this problem. The approach is similar to the prototype solution developed by Kevin.
It could accommodate both batch jobs and real-time updates. End users would communicate with the backend via /generate endpoint to enqueue new TTS jobs or via /audio backend to retrieve path to the generated TTS files.

Happy update!

Update on current state of things.

Small status update from debugging efforts.

To speak on enabling gRPC for ISVC, our plan would be to use the Kserve's V2 Inference Protocol, which supports both gRPC and HTTP/REST interfaces. Currently, all our services were built with V1 protocol in mind, which only supports HTTP/REST interface.

Weekly Update

What are you doing with the threshold argument? Are you late filtering the response from the inference service, or invoking the service with a threshold as the constraint? If the latter, is there any reason you couldn't late filter a cached response (i.e. is the cached response somehow constrained to a limited set of thresholds)?

Okay, I've done a few not too technical sketches trying to visualize the issue we're facing.

@Joe Hoarde's HTTP API only exposes wiki_id/page_id/revision_id parameters, which would cover the use-case for the Mobile Apps team. However, our service also exposes additional parameters (e.g. page_title, threshold) that some users rely on. On top of that, exposing HTTP API is extremely useful for us for development/debugging.
I think those would not be as problematic if we were building a new service with hoarde in mind from the beginning, however we're trying to integrate caching into existing services.

I want to share a small update from our side on where we are.

I see the regime with >10s p99 latencies, however it happened during the night and not during running those tests. It seems to me that the Grafana numbers aligns well with the reported latencies above see:

1. page_id + lang requests: https://grafana.wikimedia.org/goto/cfgzhd4aveg3kf?orgId=1
2. page_title + lang requests: https://grafana.wikimedia.org/goto/ffgzhfegn63uoc?orgId=1
3. page_id + lang + revision_id requests: https://grafana.wikimedia.org/goto/bfgzhglvqmpdse?orgId=1

When investigating T420931, I found that my custom async load test script achieves >300 RPS against the same service with 5 replicas, whereas the locust test against 1 replica reports only ~0.67 RPS. The discrepancy comes down to the Locust configuration:

I'm sharing load test numbers tested against production deployment on eqiad using internal endpoint. I've made sure the responses return valid predictions and I ran the load test after a few hours of cooldown to make results are not skewed by caching on the MWAPI side.

@Isaac The details of the cache and how exactly will it be implemented to Article Topics is still not fully decided. Current approaches we explored would work with page_id, whereas page_title requests would not go through cache. This ticket does not take cache into consideration, but we're verifying how fast can we get without cache. As a bonus, I can also check the page_title variant in this ticket so we'll have more context on it :)

After lowering the maximum input token length to 4096, we seem to be able to process all incoming requests. I will figure out optimizations we could make to allow bigger input lengths, but the current 4096 token limit should already be good enough for testing our policies.

If I understand you correctly (and if I don't, please don't hesitate to correct me), you're arguing that we might have uses that can't be satisfied, which would force a product team to build an HTTP API to serve them, one that would otherwise have also worked as the lambda (while providing an example of a hypothetical use-case). Or put another way, that (a, above) we might have past use cases with extant HTTP APIs, and (b) we might have (unavoidable) future ones too.

After deployment, CoPE-A-9B model server was successfully processing small requests of less than 500 input tokens.

The CoPE-A-9B model is now deployed on LiftWing.

Weekly Update

In T419734#11698789, @Eevans wrote:

In T414112#11698125, @elukey wrote:

In T414112#11686529, @BWojtowicz-WMF wrote:

That said, there's a practical concern on the ML Platform side and other non-ML services willing to integrate. The vast majority of our internal services communicate over HTTP, and our infra (mesh, ingress, routing) is built around that. To integrate with gRPC-only Hoarde, each service team would need to either deploy an adapter/proxy alongside their service, or deploy a gRPC-capable replica. I've prototyped the adapter approach and discussed the replica route with Luca - both are technically feasible. But as Hoarde would onboard more use-cases, I can imagine this becoming a pattern where every integrating service carries extra infrastructure just to bridge the protocol gap. So we’re shifting complexity and maintenance from Hoarde to its clients.

To me the above is the main concern, since almost every service at the foundation runs HTTP..

Is this because you a) envision the service being used to do caching for already implemented systems, b) as-yet-implemented systems that will invariably need an HTTP service anyway (and if so, why), or c) because using something other than HTTP generally imposes a burden that exceeds the benefits of using grpc (and if so, how)?

Small update on the progress.

Resolving this as this was a single time incident and the underlying concern about reference-need's high resource requests (22 CPUs, 6Gi memory) and its impact on cluster scheduling is now tracked as part of T414431, where we are optimizing resource utilization across all ISVCs.