Page MenuHomePhabricator
Paste P95140

Both HuggingFace transformers and vLLM successfully loaded jina-embeddings model on ML-Lab using MI210 GPU
ActivePublic
Actions

Authored by kevinbazira on Jul 24 2026, 8:11 AM.
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$ ls
jina_nano_retrieval_transformers.py jina_nano_retrieval_vllm.py venv
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$ python3 jina_nano_retrieval_transformers.py
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.13k/1.13k [00:00<00:00, 3.14MB/s]
configuration_jina_embeddings_v5.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 137/137 [00:00<00:00, 775kB/s]
configuration_eurobert.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 12.1k/12.1k [00:00<00:00, 29.8MB/s]
[transformers] A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-embeddings-v5-text-nano:
- configuration_eurobert.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
[transformers] A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-embeddings-v5-text-nano:
- configuration_jina_embeddings_v5.py
- configuration_eurobert.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
/srv/venv/lib/python3.12/site-packages/torchvision/io/image.py:14: UserWarning: Failed to load image Python extension: 'Could not load this library: /srv/venv/lib/python3.12/site-packages/torchvision/image.so'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
modeling_jina_embeddings_v5.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 4.12k/4.12k [00:00<00:00, 11.6MB/s]
modeling_eurobert.py: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 48.9k/48.9k [00:00<00:00, 74.3MB/s]
[transformers] A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-embeddings-v5-text-nano:
- modeling_eurobert.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
[transformers] A new version of the following files was downloaded from https://huggingface.co/jinaai/jina-embeddings-v5-text-nano:
- modeling_jina_embeddings_v5.py
- modeling_eurobert.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 424M/424M [00:01<00:00, 358MB/s]
/srv/venv/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py:71: FutureWarning: The attention mask API under `transformers.modeling_attn_mask_utils` (`AttentionMaskConverter`) is deprecated and will be removed in Transformers v5.10. Please use the new API in `transformers.masking_utils`.
warnings.warn(DEPRECATION_MESSAGE, FutureWarning)
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 110/110 [00:00<00:00, 10944.69it/s]
Fetching 8 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 11.98it/s]
Download complete: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 54.4M/54.4M [00:00<00:00, 79.0MB/s]
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 50.6k/50.6k [00:00<00:00, 134MB/s]
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 17.2M/17.2M [00:00<00:00, 35.1MB/s]
special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 582/582 [00:00<00:00, 2.31MB/s]
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
/srv/venv/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py:202: FutureWarning: The attention mask API under `transformers.modeling_attn_mask_utils` (`AttentionMaskConverter`) is deprecated and will be removed in Transformers v5.10. Please use the new API in `transformers.masking_utils`.
warnings.warn(DEPRECATION_MESSAGE, FutureWarning)
Query embeddings: (2, 768)
Document embeddings: (3, 768)
Query: 'What is the capital of Burkina Faso?'
1. [+0.8452] Ouagadougou is the capital and largest city of Burkina Faso, with a po...
2. [+0.0684] The Great Wall of China is a series of fortifications built across the...
3. [+0.0327] Climate change has led to rising sea levels and more frequent extreme ...
Query: 'impacts of climate change on coastal cities'
1. [+0.7831] Climate change has led to rising sea levels and more frequent extreme ...
2. [+0.0742] Ouagadougou is the capital and largest city of Burkina Faso, with a po...
3. [+0.0622] The Great Wall of China is a series of fortifications built across the...
Saved embeddings to transformers_retrieval_embeddings.pt (compare with vLLM output for parity).
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$ ls
jina_nano_retrieval_transformers.py jina_nano_retrieval_vllm.py transformers_retrieval_embeddings.pt venv
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$ python3 jina_nano_retrieval_vllm.py
/srv/venv/lib/python3.12/site-packages/torchvision/io/image.py:14: UserWarning: Failed to load image Python extension: 'Could not load this library: /srv/venv/lib/python3.12/site-packages/torchvision/image.so'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Loaded prompt prefixes from repo config:
query = 'Query: '
document = 'Document: '
INFO 07-24 09:13:28 [utils.py:278] non-default args: {'trust_remote_code': True, 'dtype': 'bfloat16', 'disable_log_stats': True, 'enforce_eager': True, 'model': 'jinaai/jina-embeddings-v5-text-nano-retrieval'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
INFO 07-24 09:13:36 [config.py:876] Found sentence-transformers modules configuration.
INFO 07-24 09:13:36 [config.py:903] Found pooling configuration.
INFO 07-24 09:13:36 [model.py:942] Resolved `--runner auto` to `--runner pooling`. Pass the value explicitly to silence this message.
INFO 07-24 09:13:36 [model.py:994] Resolved `--convert auto` to `--convert embed`. Pass the value explicitly to silence this message.
INFO 07-24 09:13:36 [model.py:617] Resolved architecture: TransformersEmbeddingModel
INFO 07-24 09:13:36 [model.py:1752] Using max model len 8192
INFO 07-24 09:13:36 [vllm.py:977] Asynchronous scheduling is disabled.
WARNING 07-24 09:13:36 [vllm.py:1033] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
WARNING 07-24 09:13:36 [vllm.py:1058] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
INFO 07-24 09:13:37 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
INFO 07-24 09:13:37 [vllm.py:1234] Cudagraph is disabled under eager mode
INFO 07-24 09:13:37 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
WARNING 07-24 09:13:40 [system_utils.py:157] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
/srv/venv/lib/python3.12/site-packages/torchvision/io/image.py:14: UserWarning: Failed to load image Python extension: 'Could not load this library: /srv/venv/lib/python3.12/site-packages/torchvision/image.so'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
(EngineCore pid=1761) INFO 07-24 09:13:47 [core.py:112] Initializing a V1 LLM engine (v0.22.1) with config: model='jinaai/jina-embeddings-v5-text-nano-retrieval', speculative_config=None, tokenizer='jinaai/jina-embeddings-v5-text-nano-retrieval', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=jinaai/jina-embeddings-v5-text-nano-retrieval, enable_prefix_caching=False, enable_chunked_prefill=False, pooler_config=PoolerConfig(task=None, pooling_type=None, seq_pooling_type='LAST', tok_pooling_type='ALL', use_activation=True, dimensions=None, enable_chunked_processing=False, max_embed_len=None, logit_mean=None, logit_sigma=None, logit_bias=None, logit_scale=None, step_tag_id=None, returned_token_ids=None), compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=1761) INFO 07-24 09:13:48 [parallel_state.py:1422] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.64.161.5:33911 backend=nccl
(EngineCore pid=1761) INFO 07-24 09:13:48 [parallel_state.py:1735] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=1761) INFO 07-24 09:13:49 [gpu_model_runner.py:5037] Starting to load model jinaai/jina-embeddings-v5-text-nano-retrieval...
(EngineCore pid=1761) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(EngineCore pid=1761) WARNING 07-24 09:13:49 [utils.py:192] TransformersEmbeddingModel has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
(EngineCore pid=1761) INFO 07-24 09:13:49 [base.py:117] Using Transformers modeling backend.
(EngineCore pid=1761) /srv/venv/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py:71: FutureWarning: The attention mask API under `transformers.modeling_attn_mask_utils` (`AttentionMaskConverter`) is deprecated and will be removed in Transformers v5.10. Please use the new API in `transformers.masking_utils`.
(EngineCore pid=1761) warnings.warn(DEPRECATION_MESSAGE, FutureWarning)
(EngineCore pid=1761) INFO 07-24 09:13:49 [rocm.py:552] Found incompatible backend(s) [TURBOQUANT] with AttentionType.ENCODER_ONLY. Overriding with ROCM_ATTN out of potential backends: ['ROCM_ATTN', 'TRITON_ATTN'].
(EngineCore pid=1761) WARNING 07-24 09:13:49 [compilation.py:1303] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
(EngineCore pid=1761) INFO 07-24 09:13:49 [weight_utils.py:647] No model.safetensors.index.json found in remote.
(EngineCore pid=1761) INFO 07-24 09:13:49 [weight_utils.py:922] Filesystem type for checkpoints: XFS. Checkpoint size: 0.39 GiB. Available RAM: 368.44 GiB.
(EngineCore pid=1761) INFO 07-24 09:13:49 [weight_utils.py:945] Auto-prefetch is disabled because the filesystem (XFS) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.87it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.87it/s]
(EngineCore pid=1761)
(EngineCore pid=1761) INFO 07-24 09:13:50 [default_loader.py:397] Loading weights took 0.37 seconds
(EngineCore pid=1761) INFO 07-24 09:13:50 [gpu_model_runner.py:5132] Model loading took 0.4 GiB memory and 0.998819 seconds
(EngineCore pid=1761) INFO 07-24 09:13:51 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=1761) INFO 07-24 09:13:51 [core.py:309] init engine (profile, create kv cache, warmup model) took 0.75 s
(EngineCore pid=1761) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
(EngineCore pid=1761) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
(EngineCore pid=1761) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
(EngineCore pid=1761) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'factor'}
(EngineCore pid=1761) INFO 07-24 09:13:53 [vllm.py:977] Asynchronous scheduling is disabled.
(EngineCore pid=1761) WARNING 07-24 09:13:53 [vllm.py:1033] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(EngineCore pid=1761) WARNING 07-24 09:13:53 [vllm.py:1058] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(EngineCore pid=1761) INFO 07-24 09:13:53 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(EngineCore pid=1761) INFO 07-24 09:13:53 [vllm.py:1234] Cudagraph is disabled under eager mode
(EngineCore pid=1761) INFO 07-24 09:13:53 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
INFO 07-24 09:13:53 [offline.py:41] Supported pooling task: embed
INFO 07-24 09:13:53 [io_processor.py:61] Loaded prompt prefixes for input_type: ['query', 'document']
Processed prompts: 0%| | 0/2 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](EngineCore pid=1761) WARNING 07-24 09:13:54 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
Processed prompts: 100%|███████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2.85it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 103.60it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Query embeddings: (2, 768)
Document embeddings: (3, 768)
Query: 'What is the capital of Burkina Faso?'
1. [+0.8449] Ouagadougou is the capital and largest city of Burkina Faso, with a po...
2. [+0.0690] The Great Wall of China is a series of fortifications built across the...
3. [+0.0332] Climate change has led to rising sea levels and more frequent extreme ...
Query: 'impacts of climate change on coastal cities'
1. [+0.7827] Climate change has led to rising sea levels and more frequent extreme ...
2. [+0.0752] Ouagadougou is the capital and largest city of Burkina Faso, with a po...
3. [+0.0640] The Great Wall of China is a series of fortifications built across the...
Parity vs transformers backend (cosine per input, expect ~1.0):
queries: ['0.99991', '0.99995']
documents: ['0.99994', '0.99995', '0.99994']
(EngineCore pid=1761) INFO 07-24 09:13:54 [core.py:1266] Shutdown initiated (timeout=0)
(EngineCore pid=1761) INFO 07-24 09:13:54 [core.py:1289] Shutdown complete
somebody@ml-lab1002:~$
somebody@ml-lab1002:~$
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits