I've talked with @kostajh and confirmed that the following SLOs are ok for now so we can proceed with next steps

Availability SLI: percentage of all requests that receive a non-error (non-5xx) response. SLO: 95%
Latency SLI: percentage of successful (2xx) requests that complete within 2000 ms. SLO: 90%

I wonder if we're already hitting the CPU-GPU data transfer bottleneck with longer inputs,

I think this is what the task wants to address, or to put it better see if it is an issue and improve it.

though the short-input test was run in ML-Lab while this one is on Lift Wing.

both of these tests were ran on ml-lab.

I've tracked down what was causing the above failure or to be more precise what avoids it:
With prefix caching on, vLLM 0.22.1 used an experimental cache mode for clef-flash's linear-attention layers. That mode crashes the vLLM engine when two or more long requests are processed at the same time and after that, every request failed with EngineDeadError.
The previous mL-lab load tests didn't catch it because they sent either many short requests or one long request at a time. Lift Wing traffic was the first to send several long ones at once.
I've replicated the above issue on ml-lab when sending parallel requests with 12k input tokens and also verified that the attached patch fixes this.

The vllm deployment failed so I've reverted it for now to the previous state

In T440223#12423158, @Fuzheado wrote:

Thanks @isarantopoulos for doing this. Right now the Wikitech page lists it in the "public API and Studio" section, but it seems this is only available as a predict API endpoint so it's not usable in liftwing-studio. Maybe change that on the page, or make a new set of columns for each model showing whether it's available as API call or in studio.

I've added a vllm backend to the qwen model that clef-flash is based on in order to speed up the service. Below are the load tests with the new variant as test on ml-lab.
Locust with 8 concurrent users

Type  Name                                  # reqs    # fails  Avg  Min  Max  Med  req/s  failures/s
----  ------------------------------------  ------  ---------  ---  ---  ---  ---  -----  ----------
POST  /v1/models/clef-flash:predict [text]    3226  0 (0.00%)  297   86  695  300  26.86        0.00
----  ------------------------------------  ------  ---------  ---  ---  ---  ---  -----  ----------
      Aggregated                              3226  0 (0.00%)  297   86  695  300  26.86        0.00

@BWojtowicz-WMF sharing here a modified version of your opencode config (includes only the providers.

Thanks for all the work!
I added a row in the list of available models on Wikitech https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Large_Language_Models#Available_Models
Since this is using a huggingface backend and not vllm we don't expect it to be that performant but it is fast enough to allow experimentation, We'll follow up with load tests.
Also note that the frameworks we're using (kserve, vllm etc) have note yet adopted the endpoints that decision models have introduced (v1/systemone & v1/decisions) so here we're using a standard kserve :predict endpoint for now.

Thanks for returning to this @santhosh. Is this model still the best candidate for low resource languages or are there other alternatives?
We can offer a first deployment of this model for some initial explorations and we can later assess its performance and work on any improvements if needed.

We've started some work on the SLOs for the following services which are also sorted by priority (highest prio first):

WIP SLO draft is currently being worked on here https://wikitech.wikimedia.org/wiki/SLO/LiftWing_LLM

Resolving this for now as there was no follow up action and it has not occurred again.

This work has resulted in the setup available under the directory https://gitlab.wikimedia.org/repos/machine-learning/liftwing-studio/-/tree/main/experiments/litellm?ref_type=heads
(LibreChat + LiteLLM)

Thanks for creating this task Fabian. We've been looking into decision models as well and there is indeed a lot of benefit in using them. We'll get started on this soon with clef-flash as the first candidate to try which would be easier to scale because of its size. Does that sound ok to you or is there a strong preference for another model?

Would we (ML) need to make any change or is this purely on LAC side?
As far as I know only the Apps team is using the article topics hoarde endpoint. @Dbrant would this change affect you at all?

One thing that needs further discussion is how to define the SLOs to take concurrent requests into consideration in the availability SLI. When designing an SLO, the SLIs that are defined need to be things that we control and in this case it won't be true as both TTFP and TPOT would be affected. I think this should also be defined in the services in the vllm parameter MAX_NUM_SEQS.

Legacy :predict path:

@kevinbazira Thanks for all the tests! regarding the missing :predict endpoints; I think this makes sense since there is no code that defines this path here. I propose that we only offer /chat/completions endpoints for LLMs and have a single API instead of maintaining both predict and /chat/completions. What do you think?

So are 1 hour TTLs sufficiently long for this requirement @isarantopoulos?
If you're OK with the current settings, then we might be done here and can maybe resolve the ticket.

During this experiment we will be serving a snapshot of data which means audio files based on specific revisions. No new data is generated so if we can have a higher TTL that would be great, otherwise I don't believe it is going to be an issue.

The total size is much smaller than the initial request since we are only generating audio for the Lead section of each article. @kevinbazira can provide more info either here or in the related task T436758: Publish TTS v1 audio files with standard caching at the CDN

@ssingh o/
Just a heads up that @kevinbazira has started generating the batch of audio files that will be used in the TTS experiment that will start on Monday the 5th of October.

Since were generating the audio files I'm marking this task as resolved

Note that gpt-oss-safeguard-model has been moved in the task T439395: Expose gpt-oss-safeguard-20b publicly

o/ @pfischer. We would like to discuss and negotiate the SLOs defined in this task in order to reach a consensus. Let us know what you think.

A follow up item before we close this task is for the ML team to add an explicit statement in the documentation on Wikitech that these models exist in eqiad https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Large_Language_Models. We'll add that and resolve it afterwards.