Summary
Deploy the Jina Embeddings v5 text nano (retrieval) model as an InferenceService on Lift Wing, served via vLLM in the llm namespace, alongside the existing Qwen3 embeddings service. The model produces 768-dimensional embeddings exposed through a KServe / OpenAI-compatible predict API.
- Model: https://huggingface.co/jinaai/jina-embeddings-v5-text-nano
- vLLM/retrieval variant: https://huggingface.co/jinaai/jina-embeddings-v5-text-nano-retrieval
The service is served through the shared embeddings model server (extended for vLLM pooling on the jina-embeddings branch of inference-services) and deployed via the kserve-inference chart, mirroring the existing embeddings / embeddings-staging isvcs.
Technical notes
- We have recently created an initial implementation on the jina-embeddings branch of inference-services: the embeddings model server (src/models/embeddings/model_server/model.py) has a vLLM pooling code path (VLLM_RUNNER=pooling, POOLING_TYPE=LAST, TRUST_REMOTE_CODE=True, encode(pooling_task="embed")), plus a jina-embeddings docker-compose service and README docs. The implementation confirmed to be working locally for both qwen3 and jina embeddings.
- Dependency requirement: Jina v5 requires transformers>=5.0.0. The embeddings service requirements.txt already pins transformers==5.0.0 (and kserve==0.16.0); the base image provides vLLM 0.14. Any base-image or dependency bump must preserve transformers>=5.0.0.
- License risk (potential blocker): the retrieval variant is CC BY-NC 4.0 (non-commercial); the base model is Apache-2.0. WMF commercial-use terms must be confirmed with Legal before a production deploy. to be confirmed
- Base image risk: the production Blubber image is AMD vLLM 0.14; Jina v5 docs reference newer vLLM. If model load fails on architecture/API mismatch, escalate to a newer amd-vllm base image. The AMD image cannot run on a Mac without an AMD GPU.
- Deployment follows the existing embeddings pattern in helmfile.d/ml-services/llm/values-ml-serve-eqiad.yaml (image, STORAGE_URI from wmf-ml-models S3, env vars, 1× AMD GPU, 6 CPU / 16Gi, node affinity + mi300x-experiments tolerations).
- Monitoring is already enabled namespace-wide (monitoring.enabled: true → prometheus scrape on :8080/metrics); the new isvc inherits it, so no new monitoring config is expected.
- Load testing builds on the existing test/locust/models/embeddings/embeddings_test.py (currently targets /v1/models/qwen3-embedding:predict).
- Removing qwen3 embeddings is out of scope. We can remove it once Search Team stops using it. It will enable us to be more flexible with the deployments.
- Check with the team if we need something special for alerting.
- Requirements Check with the team if we already have a docker image that satisfies the requirements below;
Our current images are behind of all dependencies above.
Acceptance criteria
- jina-embeddings branch of inference-services finalized, service builds and serves locally/on a GPU host, and merged; new embeddings image published by PipelineBot.
- Model weights uploaded to the wmf-ml-models S3 bucket with a recorded STORAGE_URI/version path.
- jina-embeddings and jina-embeddings-staging isvc entries added to values-ml-serve-eqiad.yaml, deployed to staging, and reaching Ready.
- New isvc appears in Prometheus/Grafana Lift Wing dashboards. (as enabled by default)
- Locust load test added under test/locust/models/embeddings/ targeting the jina endpoint, with a captured latency/RPS baseline.
- Run locust tests and compare with the qwen3 embeddings results.
- Production deployment
- Update documentation: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Large_Language_Models
