qwen36-27b: MAX_NUM_BATCHED_TOKENS experiments with MAX_NUM_SEQS=32

Plan: experiments for the qwen36-27b and qwen38-27b SLOs

Ah, yes, this is what I understood, but the diagram shows "Orchestrator (e.g. Change-prop)" emitting the page_revert_risk_wikidata_prediction_change to eventgate -> kafka.

I think the arrows are kinda misleading in the schema. We just mean that the change-prop calls the LiftWing and goes on...

In T433699#12431388, @Ottomata wrote:

Thanks! How does change_prop emit the mediawiki.page_revert_risk_wikidata_prediction_change.v1 event?

A quick heads-up following our ML meeting: we decided to go with the following schema and proceed with the Hoard integration.

Tested revertrisk-wikidata on ml-staging-codfw from this liftwing-patch and this deployment_patch and grpc_deployment using rev_id 1892513445:

Here are the grafana snapshots: Istio-Qwen36-27b and ML-Infra-qwen36-27b.

These are the results of the vllm-bench (latencies in ms).
More info in the paste.

Thank you @hashar for your comment.

Latest Version of reverence-risk model deployed

Next steps:

1. Deploy the latest version of the reference-risk model server: patch-1351300
2. Deploy the latest changes in the liftwing-openapi-server: patch-1351302

Alert: Oct 1, 2026, 7:22 PM
description = The rate of errors (non-2/3/4xx return codes) in the LiftWing service llm-qwen3-14b-predictor in eqiad is higher than the configured threshold.

Sharing the 96GB partition size of the ml-serve1013/1014 machines.

Qwen3-14b Deployed on ml-serve1015

Starting gradually to move the LLMs one by one to the suggested machines.

1. llm-qwen3-14b -> ml-serve1015
2. llm-qwen36-27b -> ml-serve1013/14
3. llm-qwen38-27b -> ml-serve1013/14
4. cope-b-a4b -> ml-serve1013/14

After the MR was merged I deployed the Qwen3.8 model on LiftWing studio.
You can check this in the image below:

After increasing the cpu quotas in llm namespace by merging/deploying this patch: 1343012
The llm-qwen38-27b service is working properly.

1. Qwen3.8-27B deployed to llm namespace.

Hey @Ottomata, thnx for your comment.

In T433699#12304464, @Ottomata wrote:

Q: I wonder if a better generalized route would be like:
Public LiftWing HTTP API -> LAC -> LiftWing Lambda
And if needed, LiftWing HTTP API could add support for passing Cache-Control: no-cache through to LAC.

The vllm-bench results can be found in the following paste:

I created a subtask for the benchmark: https://phabricator.wikimedia.org/T437034 adding my findings there.

Hey, just a short question regarding LiftWing LLM testing:
I see that the current task is an old one (Oct 15 2024) and it refers to https://gitlab.wikimedia.org/repos/research/llm_evaluation/-/blob/mnz/llmperf/llmperf/results/huggingface.ipynb?ref_type=heads .

The latest changes on LiftWing-ApenAPI-Specs are deployed.

I developed the LLM chat-completions endpoint for the llm-* models (currently llm-qwen3-14b, llm-qwen36-27b): /service/lw/inference/v1/models/llm-<model>/openai/v1/chat/completions

Hey @elukey thank you very much for your comment.

Hi @catherine.kelsey.wmde, thank you for your comment.

Update

All models are deployed on staging + prod.
Logo-detection issue is fixed merged and deployed on staging + prod, more info in the phabricator-task comment: https://phabricator.wikimedia.org/T435946#12255974

Logo-detection model CI/CD failure is fixed.
Merged and deployed on staging + prod.

I created this related task https://phabricator.wikimedia.org/T435946 regarding the CI/CD failure on the logo-detection model server. The fix is in this patch
All the updates about it will be pasted in the corresponding task.

I removed old edit-check. I just left the latest ones.

The latest changes following the updates on the descriptions for the editing-suggestions are merged and deployed on staging and prod.
The updated csv file with the latest descriptions can be found here
Here are the pods running:

Excluding bots lowered the base rate but the model discriminates far better on the human-editor population (AUC 0.72 → 0.89).

Thank you for your comment @isarantopoulos.

Just chiming in as I've noticed a discrepancy in how the classes should be configured. In damaging models classes maybebad, likelybad and verylikelybad are not mutually exclusive and demonstrate relaxed versions of the same filter. This means that their upper bound is always a probability of 1. More references of this can be found in mediawiki-config repo

Following review feedback, three fixes were applied to the ground truth and the comparison was made valid across models.

Thank you @achou for your comment it was very helpful.
I am pasting here some initial results.

I suspect it happens often, since method includes rollback, which reverts every consecutive edit by the last editor. And the ones we'd miss are the later edits in a rolled-back run, likely the >higher-scoring ones, so they'd end up labelled as negatives right where we're defining verylikelybad. Can you make a quick check? something like COUNT(*) WHERE rev_reverted_newest_id != rev_reverted_oldest_id. If it's non-trivial, the > fix should be easy — oldest -> newest is a contiguous run on one page, so we can join on page.page_id with rev_id BETWEEN oldest AND newest instead of equality.