Plan: ml-serve1015 baseline benchmark (before/after MI300X tuning)

Thanks for the suggestion! I like the idea of keeping tests next to the model code, but there are a few things we'd need to consider:

Hm. I was imagining just hiding LAC behind whatever endpoint you expect LiftWing HTTP callers to call.
Where possible, LiftWing inference calls would be via gRPC lambda behind LAC. There'd be no route to the inference endpoint without LAC.

Thanks! That makes sense.

In T433699#12304464, @Ottomata wrote:

Wikimedia Enterprise — switch to the LAC endpoint

Q: I wonder if a better generalized route would be like:

Public LiftWing HTTP API -> LAC -> LiftWing Lambda

And if needed, LiftWing HTTP API could add support for passing Cache-Control: no-cache through to LAC.

We've also settled on the name for the user-facing UI: Hummingbird Chat

We decided to move forward with LibreChat for LiftWing Studio.

@DPogorzelski-WMF Could you also update the doc https://wikitech.wikimedia.org/wiki/Machine_Learning/AMD_GPU to reflect the current GPU partitioning when you get a chance? Thanks!

Here are the main findings from comparing Open WebUI and LibreChat across a few areas: user-facing evaluation, agent/workspace setups, and RAG. The two frameworks overlap a lot in core capabilities, but their product direction and user experience feel a bit different.

Here are a few areas I think are worth comparing more closely:

• MCP / agent workflows
• Access control and LiteLLM integration
• User-facing experimentation / evaluation workflows

I spent some time looking into LiteLLM's Agent and MCP gateways today, mainly to understand where they fit and clarify some of the questions we had around whether LiteLLM sits before or after the harness and tool servers.

Some possible next steps from today's standup:

I also tried the LibreChat Agent setup today, using the same task from llm-lab experiment 05.

@achou would it be relatively easy to add to this list, or would it be impossible to add disallow suggestions in sections that are likely "plot summary" related?

In T424923#12285385, @Gehel wrote:

Machine-Learning-Team will give the list of partitions / nodes to repartition

Thanks @DPogorzelski-WMF, this is really helpful. That clears up the infra-side questions I had.

Hi, just wanted to clarify: the pilot wikis already released are en, fr, ar, and pt, and the new releases (part of this task) are ru, ja, nl, zh, de, and es, right?

Regarding Part 2: Scaling to more languages, the Growth team would like to enable the following additional languages, according to T426364:

Deployed 2026-08-17, verified with kubectl describe quota quota-compute-resources. Resolving.

Reopening: @elukey asked on #talk-to-machine-learning for a rebuild of the ores-legacy image (debmonitor-client fails to install on it) — same root cause as this task, and ores-legacy isn't the only one left.

We're planning to taint every GPU pool so a service can only land on the GPU type it explicitly tolerates (details above). Before we ask for the taints to be applied, a few things we'd like SRE's read on — most sit on the infrastructure side.

Thanks for working on this, George!

The dashboard shows the frequency and latency of the 500 errors: https://grafana.wikimedia.org/d/zsdYRV7Vk/istio-sidecar?orgId=1&var-cluster=aWotKxQMz&var-namespace=revertrisk&var-backend=revertrisk-wikidata-predictor-00005-private&var-response_code=500&var-quantile=0.5&var-quantile=0.95&var-quantile=0.99&from=now-24h&to=now&timezone=utc

Update — added the last two pieces:

Mitigation deployed today: scaled revertrisk-wikidata to 3 replicas, following the ChangeProp concurrency reduction. Monitoring Grafana, latency trending back toward pre-event-stream baseline.

Baseline established and tooling landed (see benchmarks/). Public-endpoint half done in T432830. Closing — Locust deferred as noted in benchmarks/README.md.

Public-endpoint baselines run from Toolforge for both models, results and overhead comparison posted above. Wrapper fix folded into CL 1319848. Closing.

While testing the rollout in T420883, I found another deployment constraint that's related to this discussion.

Enterprise reported a latency regression after the event stream went live on 2026-07-15. Since the event-stream workload and Enterprise traffic share the same deployment, I started investigating a short-term mitigation by increasing revertrisk-wikidata from 2 -> 3 replicas.

• Qwen3-14B FP8 (including FP8 KV cache) is now deployed, on a single 24 GB GPU partition. (T431852)
• Benchmarking baseline recorded for Qwen3-14B FP8 (T431851#12145547).
• Public-API benchmarking baseline recorded for both public models, run from Toolforge (T432830#12154062).
• Documentation: added the Wikimania program page, a volunteer/developer quick-start extracted from the main docs.

Ran the public endpoint (api.wikimedia.org) from Toolforge, which bypasses the 100/hour cap via the known-network tier. Same fixed config as the internal baseline, so the numbers compare directly.

@OKarakaya-WMF thank you! totally agree, your conclusion is spot on: this setup gets impacted by load faster than the 27B. One thing: could you add this run's UTC window + a dashboard link scoped to that window to the results above (like the 27B post)? That way anyone can open the live panels for the exact run instead of just the screenshot.

Thanks @OKarakaya-WMF! Quick note — this looks like a single vllm-bench call at concurrency 32, so it prints vllm-bench's per-run output but not the combined summary table. That table (and the full 1/8/32/64 sweep) comes from the benchmark_llm.sh wrapper. You can get the sweep + table for the 14B with:

Most services are rebuilt and deployed to prod: outlink, article-country, revscoring, revertrisk-multilingual, revertrisk-wikidata, and editing_suggestions.

New program page for Wikimania is created: LiftWing/Large_Language_Models/Wikimania_2026

I'm wrapping up the Available Models / Deployment sections of the LLM docs. I pulled precision, tensor-parallel, and context length from the deployment configs, but need to confirm a few base-model identities:

• Qwen3-27B FP8 (including FP8 KV cache) is now deployed (T431978).
• Initial benchmarking baseline for Qwen3-27B FP8 has been recorded (T431851#12132189).
• The V0 LLM documentation is now up to date on LiftWing/Large Language Models; next steps are tracked in T431847#12127578.

V0 benchmark baseline — llm-qwen36-27b (FP8)

Thanks @OKarakaya-WMF. Yes, I'm passing --percentile-metrics ttft,tpot,itl,e2el. On why I switched to vllm-bench (the standalone Rust binary) rather than the Python vllm bench serve: it wasn't about the metrics, both support the e2el flag. I needed a custom Host header to route through the internal Knative gateway, which the vllm bench serve build I had didn't expose. Plus, it's much lighter to install, a single binary, no Python env or the full vllm/torch dependency stack. Its JSON schema matches vllm bench serve, so results stay comparable.

Got a working benchmarking setup on the ML-lab host, against llm-qwen36-27b on the internal endpoint; smoke test passes. Full concurrency sweep + baseline numbers still to come.

Update: V0 documentation is live

The revertrisk-wikidata prediction stream is now live in production and available via EventStreams:
https://stream.wikimedia.org/v2/ui/#/?streams=mediawiki.page_revert_risk_wikidata_prediction_change.v1

Update: proceeding to production.