I'm closing this task as we have deployed Jina Nano embeddings to liftwing successfully.

works well in the grafana dashboard:
https://grafana-rw.wikimedia.org/goto/dfu227cq03wn4e?orgId=default

attention sdpa after TORCH_BLAS_PREFER_HIPBLASLT=0 fix.

attention eager after TORCH_BLAS_PREFER_HIPBLASLT=0 fix.

this does not seem to be a memory issue as peak values are low and free memory is high.

confirmed that the high performance results are from ml-serve1012 (192GB).
The latency increases to ~5 seconds (not acceptable for prod) when we place the app to another node (24GB).

I've removed ml-serve1012 and interesting that the response time has increased to ~5 seconds. I'll set ml-serve1012 to see if this is the issue.

response_time: 5.714561s
ozge@stat1010:~$ curl "https://inference.svc.eqiad.wmnet:30443/v1/models/jina-embeddings:predict"   -X POST   -d '{"input": ["hello world"]}'   -H "Host: jina-embeddings.llm.wikimedia.org"   -H "Content-Type: application/json"   --http1.1   -w "\nresponse_time: %{time_total}s\n"

Actually, the results are much better than I expected and slightly lower than current prod.
Jina:

• Jina: 99 percentile latency is 92 ms with 95.24 req/s
• prod: 99 percentile latency is 84 ms with 98.90 req/s

prod locust latest test results to compare:
https://phabricator.wikimedia.org/T418976#11703437

I've created a patch to add missing config files.
I exclude onnx and gguf files as I think they are not need for rocm amd.

I've created a patch to deploy sentence transformers version to staging.
Can you take a look when you have time?

thank you @kevinbazira . I've addressed your comment.

I've created a patch to serve jina nano embeddings with transformers (SentenceTransformer).
Can you take a look when you have time? @kevinbazira

Flash attention2 worked after enabling it manually:

I've tried enabling flash attention2 in both mllabs and in kserve.

The static_description and the titles are updated as suggested 🎉

Hey @kevinbazira thank you for the confirmation!

thank you @achou , I've added the grafana link to my previous comment.

Possible next steps:

When I download the model, I get modeling_eurobert.py which is the transformers implementation of eurobert. I think this is the reason why we fallback to transformers and not able to use vllm features.

Updated as follows with a small typo fix in npov.

two patches for review. We can deploy it on Monday:

I've tested with enforce eager = false.
Now the latency is consistent with ~50 miliseconds.
This is better than before although still slower than my local cpu with ~5 miliseconds.

Jino Nano is a EuroBERT based model and it falls back to transformers at server start-up and does not use vllm features.

jina small is supported via vllm but jina nano is not supported:

jina nano model is based on EuroBert and it falls back to transformers (Resolved architecture: TransformersEmbeddingModel).
jina small is official supported in vllm.

this time the server started successfully but the inference is sometimes slow (less than a second to 30 seconds.)
looking into a possible solution.

staging deployment failed with ProgressDeadlineExceeded

Awesome, thank you @DLynch . I think it's a great idea.

cool I'm adding llm-qwen3-14b results below:

I see we have hit to 100% gpu usage during the load test. I think we can consider to improve it based on requirements in later phases.

sharing lm-qwen3-14b results with the following command (Maximum request concurrency: 32):

It looks awesome! Thank you very much.

deployment completed. We see a good amount of decrease in latency. We can re-run the load tests to measure the difference more reliably.

got No available memory for the cache blocks error. tuning some parameters.

thank you @achou for working on this.
adding a small note:
vllm bench serve can return End-to-End Latency metrics by passing --percentile-metrics ttft,tpot,itl,e2el .
sharing a full example in T431554

We processed 30K articles in total based on their edit counts by using gpt-oss-120b model and filtered based on quality by using gemma4 model.
After filtering, we kept suggestions from 14K of the articles which yields 29350 suggestions in total from 3 different categories namely MOS:GEO, NPOV and simplify language.
We serve the suggestion publicly by using an api.

• T416869 Edit Suggestions - Dataset for hackathon : We started with running an hackathon on a pre-defined dataset for consistency and diversity of the articles.
• First we started with wikitext but due to the formatting issues, we decided to use html of the articles.
• T419840 Edit Suggestions - Formatting for html to text: We solved issues related to converting html to human readable text for context.
• T418097 Edit Suggestions - Edit suggestion generation with loose edit types: We experimented with the loose edit types and decided to continue with pre-defined edit types as it gives more control over the generated sugestions. We have also experimented with different prompts e.g. simple prompt and complex prompt.
• Creating a list of most frequent user edits in enwiki: We use this as a base of target edit types. We enriched it with wished edit types by community and teams. We defined the priority on this list with the help of Editing Team.
• T418102 Edit Suggestions - Edit suggestion generation with pre-defined edit types: We experimented with pre-defined edit types.
• T424347 Editing Suggestions - Dataset generation logic from edits: We created a fixed dataset per edit type for evaluation.
• T421672 Edit suggestions - Precision recall on grammar edit suggestions based on edits: We evaluated edit types by using user edits and llm-as-a-judge. This helps us to identify which models are better, which edit types are more suitable and how the context (article, paragraph) and prompt should be shaped.
• T425957 Editing Suggestions - Eval pipeline for enwiki: Converting evalution to scripts that we can run on the mllab machines.
• T429470 Editing Suggestions - Phase I dataset: Assessing which dataset fits best for the experiment on production. We decided to create fresh suggestions.
• T428882 Editing Suggestions - Increase/refresh LLM generated suggestions: Increase the number of total suggestions to ~30K. This also includes the analysis about how many suggestions will be enough for the experiment. We also analysed which articles should be picked based on their yearly and total edit counts. We generated suggestions with one model and assessed their quality with another model by using llm-as-a-judge. We used llm-as-a-judge for filtering.
• T427794 Editing Suggestions - api: We created an api for serving suggestions including deployment. This enables us to experiment quickly with different suggestion types without requiring an investment on infrastructure. Noting that we have discussed and agreed on how we integrate (api rather than cassandra and open search) and the schema with the Editing Team.
• T424988 Editing Suggestions - Multi-language support: We explore identifying edit types in different wikis. We started working on dewiki and NPOV edit type as well but we haven't finished this task as it will be followed in another goal.
• T430029 Editing Suggestions - Updating documentation.

1. vLLM bench serve

hey @Sucheta-Salgaonkar-WMF , we have finished the tasks on MLTeam side for this goal. I see there are a couple of for Editing Team though. Are we in favor of closing this goal?

updated editing suggestions on both staging and production.

created a patch for editing suggestions

Thank you Kevin, I'm looking into editing-suggestions.

pipeline steps are implemented as script and more information added to a document:
https://gitlab.wikimedia.org/repos/machine-learning/exploratory-notebook/-/blob/main/edit_suggestions/docs/model_generated_suggestions.md?ref_type=heads

closing this ticket as multilanguage support will be handled in a new goal.

I see apps (article-country, revscoring, revertrisk-multilingual, revertrisk-wikidata) are up-to-date on prod deployment charts now.
@achou , @kevinbazira, @gkyziridis We can check together if we need to update more apps. Otherwise, we can close the task.

Deployed to prod:

Closing as the update fixes the known bracket issue.
We will separately look into the possibility of using visual editor logic for a common target validation.

Dataset and the api is updated.
Location for the file:

I think we can create a new ticket when we have the descriptions and update accordingly.
Therefore, I'm closing this ticket but please feel free to share the descriptions and we can always reopen this task e.g. in scope of bug fixing etc.