Page MenuHomePhabricator
Create Task
Maniphest T427497

Deploy CoPE-B-A4B on LiftWing
Closed, ResolvedPublic
Actions

Description

Summary

We'd like to deploy CoPE-B-A4B (https://huggingface.co/zentropi-ai/cope-b-a4b) on Lift Wing to evaluate its performance, following the deployment of CoPE-A-9B T418832: Deploy CoPE-A on LiftWing.

Technical notes

CoPE-B-A4B is fine-tuned from Gemma-4 (~25B MoE, ~52 GB BF16 weights, 256k context). Our current production vLLM 0.14 image does not support the Gemma-4 architecture, so this task depends on the vLLM ≥ 0.19 image upgrade tracked in T426766: Upgrade production vLLM image to use vLLM version >= 0.19.

Acceptance criteria

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
ml-services: deploy cope-b-a4b isvc in LW prodKevin Baziraoperations/deployment-chartsmaster+49 -49
ml-services: deploy cope-b-a4b isvc that trims response to violation, p_violation, p_safeKevin Baziraoperations/deployment-chartsmaster+1 -1
policy-violation: trim cope-b-a4b response to violation, p_violation, p_safeKevin Baziramachinelearning/liftwing/inference-servicesmain+17 -23
policy-violation: add confidence score to cope-b-a4b responsesKevin Baziramachinelearning/liftwing/inference-servicesmain+69 -6
ml-services: deploy cope-b-a4b isvc that adds confidence score to responsesKevin Baziraoperations/deployment-chartsmaster+1 -1
ml-services: deploy cope-b-a4b isvc that was migrated from HF transformers to vLLM 0.22.1Kevin Baziraoperations/deployment-chartsmaster+3 -3
policy-violation: centralize Python3.12 entrypoint and common_settings scriptsKevin Baziramachinelearning/liftwing/inference-servicesmain+6 -0
policy-violation: use custom builders to install cope-b-a4b deps in Python3.12 envKevin Baziramachinelearning/liftwing/inference-servicesmain+9 -9
policy-violation: use common_settings.sh specific to cope-b-a4b model-serverKevin Baziramachinelearning/liftwing/inference-servicesmain+20 -3
policy-violation: migrate cope-b-a4b model-server from HF transformers to vLLMKevin Baziramachinelearning/liftwing/inference-servicesmain+61 -106
ml-services: remove gpt-oss-safeguard-20b isvc from experimental nsKevin Baziraoperations/deployment-chartsmaster+0 -97
ml-services: add cope-b-a4b isvc to experimental nsKevin Baziraoperations/deployment-chartsmaster+49 -0
policy-violation: fix cope-b-a4b transformers version via PYTHONPATH orderKevin Baziramachinelearning/liftwing/inference-servicesmain+1 -1
policy-violation: containerize cope-b-a4b model-serverKevin Baziramachinelearning/liftwing/inference-servicesmain+121 -1
policy-violation: add cope-b-a4b model-serverKevin Baziramachinelearning/liftwing/inference-servicesmain+305 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptMay 28 2026, 10:18 AM
isarantopoulos added a comment.May 28 2026, 10:27 AM

The model's model card mentions Compatible with vLLM ≥ 0.20.2 for production serving so we should see if we can use that version.

kevinbazira subscribed.May 28 2026, 10:44 AM
kostajh subscribed.Jun 1 2026, 2:01 PM
kevinbazira added a parent task: T418267: Q2 FY2025-26 Goal: Host a content policy evaluation model on LiftWing.Jun 3 2026, 7:02 AM
gerritbot added a comment.Jun 3 2026, 7:09 AM

Change #1296941 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] policy-violation: add cope-b-a4b model-server

https://gerrit.wikimedia.org/r/1296941

gerritbot added a project: Patch-For-Review.Jun 3 2026, 7:09 AM
gerritbot added a comment.Jun 3 2026, 7:11 AM

Change #1296942 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] policy-violation: containerize cope-b-a4b model-server

https://gerrit.wikimedia.org/r/1296942

gerritbot added a comment.Jun 3 2026, 8:55 AM

Change #1296941 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] policy-violation: add cope-b-a4b model-server

https://gerrit.wikimedia.org/r/1296941

kevinbazira mentioned this in rMLISc9c66d9cba62: policy-violation: add cope-b-a4b model-server.Jun 3 2026, 8:55 AM
gerritbot added a comment.Jun 3 2026, 11:29 AM

Change #1296942 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] policy-violation: containerize cope-b-a4b model-server

https://gerrit.wikimedia.org/r/1296942

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2026, 11:30 AM
kevinbazira mentioned this in rMLIS282e3de5f8b3: policy-violation: containerize cope-b-a4b model-server.Jun 3 2026, 11:31 AM
gerritbot added a comment.Jun 3 2026, 2:47 PM

Change #1297166 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] policy-violation: fix cope-b-a4b transformers version via PYTHONPATH order

https://gerrit.wikimedia.org/r/1297166

gerritbot added a project: Patch-For-Review.Jun 3 2026, 2:47 PM
gerritbot added a comment.Jun 3 2026, 3:09 PM

Change #1297166 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] policy-violation: fix cope-b-a4b transformers version via PYTHONPATH order

https://gerrit.wikimedia.org/r/1297166

kevinbazira mentioned this in rMLIS149572dd98e3: policy-violation: fix cope-b-a4b transformers version via PYTHONPATH order.Jun 3 2026, 3:10 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 3 2026, 3:31 PM
gerritbot added a comment.Jun 3 2026, 5:06 PM

Change #1297199 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add cope-b-a4b isvc to experimental ns

https://gerrit.wikimedia.org/r/1297199

gerritbot added a project: Patch-For-Review.Jun 3 2026, 5:06 PM
gerritbot added a comment.Jun 4 2026, 5:03 AM

Change #1297199 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: add cope-b-a4b isvc to experimental ns

https://gerrit.wikimedia.org/r/1297199

Maintenance_bot removed a project: Patch-For-Review.Jun 4 2026, 5:30 AM
kevinbazira added a comment.Jun 4 2026, 6:30 AM

The zentropi-ai/cope-b-a4b docs show we have 3 hosting options: zentropi API, vLLM, HF transformers. Since our vLLM base image doesn't yet support cope-b-a4b as shown in: P93623, we built the model-server on HF transformers instead, which loads and runs the model successfully (see P93624).

We have deployed the cope-b-a4b LLM in the prod experimental namespace to access the MI300x GPU.

The isvc starts in the pod on eqiad as shown below:

P93837 cope-b-a4b LLM loaded successfully in prod experimental ns on MI300x GPU
1$ kubectl logs cope-b-a4b-predictor-00002-deployment-869b9cd8fc-b7znb
2+ source common_settings.sh
3+++ /usr/bin/python3 -c 'from python.resource_utils import get_cpu_count; print(get_cpu_count())'
4++ CPU_COUNT=6
5++ echo 'CPU count detected from get_cpu_count: 6'
6++ export OMP_NUM_THREADS=6
7++ OMP_NUM_THREADS=6
8++ echo 'OMP_NUM_THREADS set to: 6'
9+ MODEL_SERVER_PATH=model.py
10+ exec /usr/bin/python3 model.py
11CPU count detected from get_cpu_count: 6
12OMP_NUM_THREADS set to: 6
13INFO:root:Loading tokenizer...
14/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
15INFO:root:Loading model...
16[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
17Loading weights: 100%|██████████| 657/657 [00:21<00:00, 29.90it/s]
18INFO:root:Model loaded successfully!
19INFO:kserve:Registering model: cope-b-a4b
20INFO:kserve:Setting max asyncio worker threads as 32
21INFO:kserve:OpenAI endpoints registered
22INFO:kserve:Time series endpoints not registered
23INFO:kserve:Starting uvicorn with 1 workers
24INFO:uvicorn.error:Started server process [1]
25INFO:uvicorn.error:Waiting for application startup.
26INFO:kserve:Starting gRPC server with 4 workers
27INFO:kserve:Starting gRPC server on [::]:8081
28INFO:uvicorn.error:Application startup complete.
29INFO:uvicorn.error:Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
30INFO:uvicorn.access:127.0.0.6:50733 - "GET /metrics HTTP/1.1" 200
31INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0025026798248291016 ['http_status:200', 'http_method:GET', 'time:wall']
32INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.002496999999998195 ['http_status:200', 'http_method:GET', 'time:cpu']
33INFO:uvicorn.access:127.0.0.6:48781 - "GET /metrics HTTP/1.1" 200
34INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0045011043548583984 ['http_status:200', 'http_method:GET', 'time:wall']
35INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.004481999999995878 ['http_status:200', 'http_method:GET', 'time:cpu']
36INFO:uvicorn.access:127.0.0.6:55693 - "GET /metrics HTTP/1.1" 200
37INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0027043819427490234 ['http_status:200', 'http_method:GET', 'time:wall']
38INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0026839999999950237 ['http_status:200', 'http_method:GET', 'time:cpu']
39INFO:uvicorn.access:127.0.0.6:51877 - "GET /metrics HTTP/1.1" 200
40INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006699562072753906 ['http_status:200', 'http_method:GET', 'time:wall']
41INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.000656999999996799 ['http_status:200', 'http_method:GET', 'time:cpu']
42INFO:uvicorn.access:127.0.0.6:60305 - "GET /metrics HTTP/1.1" 200
43INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006456375122070312 ['http_status:200', 'http_method:GET', 'time:wall']
44INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006330000000005498 ['http_status:200', 'http_method:GET', 'time:cpu']
45INFO:uvicorn.access:127.0.0.6:33783 - "GET /metrics HTTP/1.1" 200
46INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0007698535919189453 ['http_status:200', 'http_method:GET', 'time:wall']
47INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0007530000000031123 ['http_status:200', 'http_method:GET', 'time:cpu']
48INFO:uvicorn.access:127.0.0.6:58125 - "GET /metrics HTTP/1.1" 200
49INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006680488586425781 ['http_status:200', 'http_method:GET', 'time:wall']
50INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006570000000039045 ['http_status:200', 'http_method:GET', 'time:cpu']
51INFO:uvicorn.access:127.0.0.6:49697 - "GET /metrics HTTP/1.1" 200
52INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006761550903320312 ['http_status:200', 'http_method:GET', 'time:wall']
53INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006619999999983861 ['http_status:200', 'http_method:GET', 'time:cpu']
54INFO:uvicorn.access:127.0.0.6:44703 - "GET /metrics HTTP/1.1" 200
55INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0009989738464355469 ['http_status:200', 'http_method:GET', 'time:wall']
56INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0009770000000060008 ['http_status:200', 'http_method:GET', 'time:cpu']
57INFO:uvicorn.access:127.0.0.6:45231 - "GET /metrics HTTP/1.1" 200
58INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006306171417236328 ['http_status:200', 'http_method:GET', 'time:wall']
59INFO:kserve.trace:kserve.io.kserve.protocol.rest.server.metrics_handler: 0.0006169999999983133 ['http_status:200', 'http_method:GET', 'time:cpu']

Querying the isvc returns:

$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/cope-b-a4b:predict" -X POST \
-d '{ "content": "CLICK HERE TO WIN $10000!!! Visit http://totallylegit.biz NOW!!!", "policy": "Content must not contain spam, phishing attempts, or deceptive links." }' \
-H  "Host: cope-b-a4b.experimental.wikimedia.org" \
-H "Content-Type: application/json" --http1.1

{"violation":1}

real	0m0.161s
user	0m0.009s
sys	0m0.005s
kevinbazira added a comment.Jun 4 2026, 7:03 AM

We have updated the liftwing_client to support this new cope-b-a4b endpoint and shared it with the PSI team to continue using it to fine-tune their policies.

kostajh added a comment.Jun 4 2026, 7:05 AM
In T427497#11983972, @kevinbazira wrote:

We have updated the liftwing_client to support this new cope-b-a4b endpoint and shared it with the PSI team to continue using it to fine-tune their policies.

thank you!

gerritbot added a comment.Jun 5 2026, 4:54 AM

Change #1297796 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: remove gpt-oss-safeguard-20b isvc from experimental ns

https://gerrit.wikimedia.org/r/1297796

gerritbot added a project: Patch-For-Review.Jun 5 2026, 4:54 AM
gerritbot added a comment.Jun 5 2026, 7:03 AM

Change #1297796 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: remove gpt-oss-safeguard-20b isvc from experimental ns

https://gerrit.wikimedia.org/r/1297796

Maintenance_bot removed a project: Patch-For-Review.Jun 5 2026, 7:31 AM
achou moved this task from Unsorted/Needs Triage to In Progress on the Machine-Learning-Team (Q1 FY2026-27) board.Jun 5 2026, 9:09 AM
kevinbazira claimed this task.Jun 5 2026, 3:45 PM
gerritbot added a comment.Jun 15 2026, 10:05 AM

Change #1302112 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] policy-violation: migrate cope-b-a4b model-server from HF transformers to vLLM

https://gerrit.wikimedia.org/r/1302112

gerritbot added a project: Patch-For-Review.Jun 15 2026, 10:05 AM
kevinbazira updated the task description. (Show Details)Jun 15 2026, 10:58 AM
gerritbot added a comment.Jun 15 2026, 11:49 AM

Change #1302112 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] policy-violation: migrate cope-b-a4b model-server from HF transformers to vLLM

https://gerrit.wikimedia.org/r/1302112

kevinbazira mentioned this in rMLIScc0cdc2e3bf5: policy-violation: migrate cope-b-a4b model-server from HF transformers to vLLM.Jun 15 2026, 11:50 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 15 2026, 12:31 PM
gerritbot added a comment.Jun 15 2026, 1:20 PM

Change #1302156 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] policy-violation: use common_settings.sh specific to cope-b-a4b model-server

https://gerrit.wikimedia.org/r/1302156

gerritbot added a project: Patch-For-Review.Jun 15 2026, 1:20 PM
gerritbot added a comment.Jun 15 2026, 1:29 PM

Change #1302156 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] policy-violation: use common_settings.sh specific to cope-b-a4b model-server

https://gerrit.wikimedia.org/r/1302156

kevinbazira mentioned this in rMLISe9c9458a23b6: policy-violation: use common_settings.sh specific to cope-b-a4b model-server.Jun 15 2026, 1:30 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 15 2026, 1:31 PM
gerritbot added a comment.Jun 15 2026, 3:46 PM

Change #1302198 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] policy-violation: use custom builders to install cope-b-a4b deps in Python3.12 env

https://gerrit.wikimedia.org/r/1302198

gerritbot added a project: Patch-For-Review.Jun 15 2026, 3:46 PM
gerritbot added a comment.Jun 15 2026, 3:49 PM

Change #1302198 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] policy-violation: use custom builders to install cope-b-a4b deps in Python3.12 env

https://gerrit.wikimedia.org/r/1302198

kevinbazira mentioned this in rMLIS93372f2669b0: policy-violation: use custom builders to install cope-b-a4b deps in Python3.12….Jun 15 2026, 3:51 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 15 2026, 4:32 PM
gerritbot added a comment.Jun 16 2026, 6:30 AM

Change #1302641 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc that was migrated from HF transformers to vLLM 0.22.1

https://gerrit.wikimedia.org/r/1302641

gerritbot added a project: Patch-For-Review.Jun 16 2026, 6:30 AM
gerritbot added a comment.Jun 16 2026, 7:03 AM

Change #1302657 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] policy-violation: centralize Python3.12 entrypoint and common_settings scripts

https://gerrit.wikimedia.org/r/1302657

gerritbot added a comment.Jun 16 2026, 8:00 AM

Change #1302657 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] policy-violation: centralize Python3.12 entrypoint and common_settings scripts

https://gerrit.wikimedia.org/r/1302657

kevinbazira mentioned this in rMLIS79dd887faa88: policy-violation: centralize Python3.12 entrypoint and common_settings scripts.Jun 16 2026, 8:01 AM
gerritbot added a comment.Jun 16 2026, 12:05 PM

Change #1302803 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] policy-violation: add confidence score to cope-b-a4b responses

https://gerrit.wikimedia.org/r/1302803

Tchanders subscribed.Jun 16 2026, 12:10 PM
gerritbot added a comment.Jun 16 2026, 12:24 PM

Change #1302641 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc that was migrated from HF transformers to vLLM 0.22.1

https://gerrit.wikimedia.org/r/1302641

gerritbot added a comment.Jun 16 2026, 1:06 PM

Change #1302803 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] policy-violation: add confidence score to cope-b-a4b responses

https://gerrit.wikimedia.org/r/1302803

kevinbazira mentioned this in rMLISd9d825c25ee5: policy-violation: add confidence score to cope-b-a4b responses.Jun 16 2026, 1:06 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 16 2026, 1:31 PM
gerritbot added a comment.Jun 16 2026, 1:36 PM

Change #1302846 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc that adds confidence score to responses

https://gerrit.wikimedia.org/r/1302846

gerritbot added a project: Patch-For-Review.Jun 16 2026, 1:36 PM
gerritbot added a comment.Jun 16 2026, 1:43 PM

Change #1302846 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc that adds confidence score to responses

https://gerrit.wikimedia.org/r/1302846

kevinbazira added a comment.Jun 16 2026, 2:17 PM

A vLLM 0.22.1 base image was published in T428577. This enabled us to migrate the cope-b-a4b model-server from HF transformers to vLLM. The latest cope-b-a4b isvc has been deployed in the prod experimental ns:

P94190 migrated cope-b-a4b model-server deployed successfully in prod experimental ns on MI300x GPU using vLLM 0.22.1
1$ kubectl logs cope-b-a4b-predictor-00001-deployment-57d5857fdd-zxkdx
2+ source common_settings.sh
3+++ /srv/venv/bin/python -c 'from python.resource_utils import get_cpu_count; print(get_cpu_count())'
4++ CPU_COUNT=6
5++ echo 'CPU count detected from get_cpu_count: 6'
6CPU count detected from get_cpu_count: 6
7OMP_NUM_THREADS set to: 6
8++ export OMP_NUM_THREADS=6
9++ OMP_NUM_THREADS=6
10++ echo 'OMP_NUM_THREADS set to: 6'
11+ MODEL_SERVER_PATH=model.py
12+ exec /srv/venv/bin/python model.py
13/srv/venv/lib/python3.12/site-packages/torchvision/io/image.py:14: UserWarning: Failed to load image Python extension: 'Could not load this library: /srv/venv/lib/python3.12/site-packages/torchvision/image.so'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
14 warn(
15INFO:root:Loading vLLM model...
16INFO 06-16 13:59:48 [utils.py:278] non-default args: {'dtype': 'bfloat16', 'max_model_len': 16384, 'disable_log_stats': True, 'model': '/mnt/models'}
17INFO 06-16 13:59:58 [model.py:617] Resolved architecture: Gemma4ForCausalLM
18INFO 06-16 13:59:58 [model.py:1752] Using max model len 16384
19INFO 06-16 13:59:59 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=16384.
20INFO 06-16 13:59:59 [config.py:100] Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512). Forcing TRITON_ATTN backend to prevent mixed-backend numerical divergence.
21INFO 06-16 13:59:59 [vllm.py:977] Asynchronous scheduling is enabled.
22INFO 06-16 13:59:59 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
23WARNING 06-16 14:00:04 [system_utils.py:157] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
24/srv/venv/lib/python3.12/site-packages/torchvision/io/image.py:14: UserWarning: Failed to load image Python extension: 'Could not load this library: /srv/venv/lib/python3.12/site-packages/torchvision/image.so'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
25 warn(
26(EngineCore pid=128) INFO 06-16 14:00:11 [core.py:112] Initializing a V1 LLM engine (v0.22.1) with config: model='/mnt/models', speculative_config=None, tokenizer='/mnt/models', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/mnt/models, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_mla_dual_rms_norm': False, 'fuse_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
27(EngineCore pid=128) INFO 06-16 14:00:13 [parallel_state.py:1422] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.67.19.34:37883 backend=nccl
28(EngineCore pid=128) INFO 06-16 14:00:13 [parallel_state.py:1735] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
29(EngineCore pid=128) INFO 06-16 14:00:13 [gpu_model_runner.py:5037] Starting to load model /mnt/models...
30(EngineCore pid=128) INFO 06-16 14:00:14 [rocm.py:507] Using TRITON_ATTN backend (selected via --attention-backend).
31(EngineCore pid=128) WARNING 06-16 14:00:14 [activation.py:349] [ROCm] PyTorch's native GELU with tanh approximation is unstable with torch.compile. For native implementation, fallback to 'none' approximation. The custom kernel implementation is unaffected.
32(EngineCore pid=128) INFO 06-16 14:00:14 [unquantized.py:285] Using TRITON Unquantized MoE backend out of potential backends: ['ROCm AITER', 'TRITON', 'BATCHED_TRITON'].
33(EngineCore pid=128) WARNING 06-16 14:00:14 [compilation.py:1303] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
34(EngineCore pid=128) INFO 06-16 14:00:14 [weight_utils.py:922] Filesystem type for checkpoints: EXT4. Checkpoint size: 47.00 GiB. Available RAM: 1405.13 GiB.
35(EngineCore pid=128) INFO 06-16 14:00:14 [weight_utils.py:945] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
36Loading safetensors checkpoint shards: 0% Completed | 0/11 [00:00<?, ?it/s]
37Loading safetensors checkpoint shards: 9% Completed | 1/11 [00:01<00:19, 1.99s/it]
38Loading safetensors checkpoint shards: 18% Completed | 2/11 [00:04<00:21, 2.37s/it]
39Loading safetensors checkpoint shards: 27% Completed | 3/11 [00:07<00:19, 2.44s/it]
40Loading safetensors checkpoint shards: 36% Completed | 4/11 [00:09<00:17, 2.49s/it]
41Loading safetensors checkpoint shards: 45% Completed | 5/11 [00:12<00:15, 2.50s/it]
42Loading safetensors checkpoint shards: 55% Completed | 6/11 [00:14<00:12, 2.53s/it]
43Loading safetensors checkpoint shards: 64% Completed | 7/11 [00:17<00:10, 2.52s/it]
44Loading safetensors checkpoint shards: 73% Completed | 8/11 [00:19<00:07, 2.54s/it]
45Loading safetensors checkpoint shards: 82% Completed | 9/11 [00:22<00:05, 2.52s/it]
46Loading safetensors checkpoint shards: 91% Completed | 10/11 [00:24<00:02, 2.54s/it]
47Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:26<00:00, 2.11s/it]
48Loading safetensors checkpoint shards: 100% Completed | 11/11 [00:26<00:00, 2.37s/it]
49(EngineCore pid=128)
50(EngineCore pid=128) INFO 06-16 14:00:40 [default_loader.py:397] Loading weights took 26.26 seconds
51(EngineCore pid=128) INFO 06-16 14:00:40 [unquantized.py:341] Using MoEPrepareAndFinalizeNoDPEPModular
52(EngineCore pid=128) INFO 06-16 14:00:41 [gpu_model_runner.py:5132] Model loading took 47.42 GiB memory and 26.679009 seconds
53(EngineCore pid=128) INFO 06-16 14:00:49 [backends.py:1089] Using cache directory: /srv/.cache/vllm/torch_compile_cache/99fcc63784/rank_0_0/backbone for vLLM's torch.compile
54(EngineCore pid=128) INFO 06-16 14:00:49 [backends.py:1148] Dynamo bytecode transform time: 7.18 s
55(EngineCore pid=128) INFO 06-16 14:00:59 [backends.py:378] Cache the graph of compile range (1, 16384) for later use
56(EngineCore pid=128) INFO 06-16 14:01:21 [backends.py:393] Compiling a graph for compile range (1, 16384) takes 32.23 s
57(EngineCore pid=128) INFO 06-16 14:01:23 [decorators.py:708] saved AOT compiled function to /srv/.cache/vllm/torch_compile_cache/torch_aot_compile/f2b68215aafdf5c86b836bde9dab705f4bd5a24e89ab6bc4933dbb9ad4391bbd/rank_0_0/model
58(EngineCore pid=128) INFO 06-16 14:01:23 [monitor.py:53] torch.compile took 41.53 s in total
59(EngineCore pid=128) WARNING 06-16 14:01:24 [fused_moe.py:1073] Using default MoE config. Performance might be sub-optimal! Config file not found at /srv/venv/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=704,device_name=AMD_Instinct_MI300X.json
60(EngineCore pid=128) INFO 06-16 14:01:26 [monitor.py:81] Initial profiling/warmup run took 2.70 s
61(EngineCore pid=128) INFO 06-16 14:01:29 [gpu_worker.py:466] Available KV cache memory: 123.23 GiB
62(EngineCore pid=128) INFO 06-16 14:01:29 [kv_cache_utils.py:1733] GPU KV cache size: 586,815 tokens
63(EngineCore pid=128) INFO 06-16 14:01:29 [kv_cache_utils.py:1734] Maximum concurrency for 16,384 tokens per request: 35.82x
64Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:06<00:00, 7.74it/s]
65Capturing CUDA graphs (decode, FULL): 100%|██████████| 51/51 [00:09<00:00, 5.27it/s]
66(EngineCore pid=128) INFO 06-16 14:01:47 [gpu_model_runner.py:6456] Graph capturing finished in 17 secs, took 3.40 GiB
67(EngineCore pid=128) INFO 06-16 14:01:47 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
68(EngineCore pid=128) INFO 06-16 14:01:47 [core.py:302] init engine (profile, create kv cache, warmup model) took 65.71 s (compilation: 41.53 s)
69(EngineCore pid=128) INFO 06-16 14:01:49 [vllm.py:977] Asynchronous scheduling is enabled.
70(EngineCore pid=128) INFO 06-16 14:01:49 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
71INFO:root:Model loaded successfully!
72INFO:kserve:Registering model: cope-b-a4b
73INFO:kserve:Setting max asyncio worker threads as 32
74INFO:kserve:OpenAI endpoints registered
75INFO:kserve:Time series endpoints not registered
76INFO:kserve:Starting uvicorn with 1 workers
77INFO:uvicorn.error:Started server process [1]
78INFO:uvicorn.error:Waiting for application startup.
79INFO:kserve:Starting gRPC server with 4 workers
80INFO:kserve:Starting gRPC server on [::]:8081
81INFO:uvicorn.error:Application startup complete.
82INFO:uvicorn.error:Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)

We also added a confidence score to cope-b-a4b inference service responses:

$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/cope-b-a4b:predict" -X POST \
-d '{ "content": "CLICK HERE TO WIN $10000!!! Visit http://totallylegit.biz NOW!!!", "policy": "Content must not contain spam, phishing attempts, or deceptive links." }' \
-H  "Host: cope-b-a4b.experimental.wikimedia.org" \
-H "Content-Type: application/json" --http1.1
{"violation":1,"p_violation":1.0,"logprob_violation":0.0,"p_safe":1.522997974471263e-8,"logprob_safe":-18.0,"confidence":1.0}
real	0m0.042s
user	0m0.011s
sys	0m0.004s
$
$
$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/cope-b-a4b:predict" -X POST \
-d '{ 
"content": "The library opens at 9am on weekdays and 10am on weekends.",
"policy": "Content must not contain spam, phishing attempts, or deceptive links."
 }' \
-H  "Host: cope-b-a4b.experimental.wikimedia.org" \
-H "Content-Type: application/json" --http1.1
{"violation":0,"p_violation":1.538173465229056e-7,"logprob_violation":-15.6875,"p_safe":0.9999998807907176,"logprob_safe":-1.1920928955078125e-7,"confidence":0.9999998807907176}
real	0m0.047s
user	0m0.010s
sys	0m0.005s
$
$
$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/cope-b-a4b:predict" -X POST \
-d '{ 
"content": "Check out my new blog where I review productivity apps. Link in my bio!",
"policy": "Content must not contain spam, phishing attempts, or deceptive links."
 }' \
-H  "Host: cope-b-a4b.experimental.wikimedia.org" \
-H "Content-Type: application/json" --http1.1
{"violation":0,"p_violation":8.939699493298122e-6,"logprob_violation":-11.625008583068848,"p_safe":0.999991059383269,"logprob_safe":-8.940656698541716e-6,"confidence":0.999991059383269}
real	0m0.045s
user	0m0.011s
sys	0m0.004s

@kostajh, the output schema has grown more keys. Please let us know whether there are fields you don't need, and we'll trim them accordingly.

Maintenance_bot removed a project: Patch-For-Review.Jun 16 2026, 2:31 PM
Tchanders added a comment.Jun 16 2026, 2:35 PM

Thanks @kevinbazira ! I think we will only need violation, p_violation and p_safe.

If we want to be as minimal as possible, we could just keep one of p_violation and p_safe, since you should be able to calculate one from the other, but I see in your first example that p_violation has lost more information in rounding than p_safe, so maybe we should keep both. Also, we may find that violation is no longer useful once we have the probabilities, but let's keep it for now until we have more confidence in the confidence scores.

We don't need logprob_violation or logprob_safe.

gerritbot added a comment.Jun 17 2026, 5:26 AM

Change #1303284 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] policy-violation: trim cope-b-a4b response to violation, p_violation, p_safe

https://gerrit.wikimedia.org/r/1303284

gerritbot added a project: Patch-For-Review.Jun 17 2026, 5:26 AM
gerritbot added a comment.Jun 17 2026, 7:50 AM

Change #1303284 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] policy-violation: trim cope-b-a4b response to violation, p_violation, p_safe

https://gerrit.wikimedia.org/r/1303284

kevinbazira mentioned this in rMLISe59414e23a5e: policy-violation: trim cope-b-a4b response to violation, p_violation, p_safe.Jun 17 2026, 7:51 AM
gerritbot added a comment.Jun 17 2026, 8:23 AM

Change #1303343 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc that trims response to violation, p_violation, p_safe

https://gerrit.wikimedia.org/r/1303343

gerritbot added a comment.Jun 17 2026, 8:35 AM

Change #1303343 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc that trims response to violation, p_violation, p_safe

https://gerrit.wikimedia.org/r/1303343

kevinbazira added a comment.Jun 17 2026, 8:59 AM

Thanks for the clarification, @Tchanders! The cope-b-a4b isvc response has been trimmed to violation, p_violation, p_safe as shown below:

$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/cope-b-a4b:predict" -X POST \
-d '{ "content": "CLICK HERE TO WIN $10000!!! Visit http://totallylegit.biz NOW!!!", "policy": "Content must not contain spam, phishing attempts, or deceptive links." }' \
-H  "Host: cope-b-a4b.experimental.wikimedia.org" \
-H "Content-Type: application/json" --http1.1
{"violation":1,"p_violation":1.0,"p_safe":1.522997974471263e-8}
real	0m0.042s
user	0m0.011s
sys	0m0.004s
$ 
$ 
$ 
$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/cope-b-a4b:predict" -X POST \
-d '{ 
"content": "The library opens at 9am on weekdays and 10am on weekends.",
"policy": "Content must not contain spam, phishing attempts, or deceptive links."
 }' \
-H  "Host: cope-b-a4b.experimental.wikimedia.org" \
-H "Content-Type: application/json" --http1.1
{"violation":0,"p_violation":1.538173465229056e-7,"p_safe":0.9999998807907176}
real	0m0.045s
user	0m0.015s
sys	0m0.000s
$ 
$ 
$ 
$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/cope-b-a4b:predict" -X POST \
-d '{ 
"content": "Check out my new blog where I review productivity apps. Link in my bio!",
"policy": "Content must not contain spam, phishing attempts, or deceptive links."
 }' \
-H  "Host: cope-b-a4b.experimental.wikimedia.org" \
-H "Content-Type: application/json" --http1.1
{"violation":0,"p_violation":8.939699493298122e-6,"p_safe":0.999991059383269}
real	0m0.045s
user	0m0.015s
sys	0m0.000s
Maintenance_bot removed a project: Patch-For-Review.Jun 17 2026, 9:31 AM
kevinbazira added a comment.Jun 18 2026, 9:51 AM

We have run load tests for the cope-b-a4b isvc, and it can handle ~32 requests/second with a median latency of ~36ms as shown below:

TypeNameRequest CountFailure CountMedian Response TimeAverage Response TimeMin Response TimeMax Response TimeAverage Content SizeRequests/sFailures/s50%66%75%80%90%95%98%99%99.9%99.99%100%
POST/v1/models/cope-b-a4b:predict383003648.3199362100804822.2367700189352041550.04274845123363.032.134387153638780.0364552587610019028074016001600
Aggregated383003648.3199362100804822.2367700189352041550.04274845123363.032.134387153638780.0364552587610019028074016001600

@Tchanders and @kostajh, please let us know whether this inference service meets your latency requirements. Once you've confirmed, we'll proceed to deploy it in LiftWing production.

kostajh added a comment.Jun 18 2026, 11:51 AM
In T427497#12032926, @kevinbazira wrote:

We have run load tests for the cope-b-a4b isvc, and it can handle ~32 requests/second with a median latency of ~36ms as shown below:

TypeNameRequest CountFailure CountMedian Response TimeAverage Response TimeMin Response TimeMax Response TimeAverage Content SizeRequests/sFailures/s50%66%75%80%90%95%98%99%99.9%99.99%100%
POST/v1/models/cope-b-a4b:predict383003648.3199362100804822.2367700189352041550.04274845123363.032.134387153638780.0364552587610019028074016001600
Aggregated383003648.3199362100804822.2367700189352041550.04274845123363.032.134387153638780.0364552587610019028074016001600

@Tchanders and @kostajh, please let us know whether this inference service meets your latency requirements. Once you've confirmed, we'll proceed to deploy it in LiftWing production.

That sounds great to me, please deploy go ahead with deploying it. Nice work!

isarantopoulos added a comment.EditedJun 18 2026, 12:02 PM

Awesome work Kevin!
It would be helpful to run load tests against the v1/chat/completions endpoint and use a prompt of 1k tokens which would give us a better understanding of real world latencies. Unless @kostajh you are already doing this on your side, in which case we can skip additional load testing for now. We plan to tackle load testing and benchmarking of LLMs as a separate workstream.

gerritbot added a comment.Mon, Jun 22, 5:47 AM

Change #1304715 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc in LW prod

https://gerrit.wikimedia.org/r/1304715

gerritbot added a project: Patch-For-Review.Mon, Jun 22, 5:47 AM
gerritbot added a comment.Mon, Jun 22, 7:48 AM

Change #1304715 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: deploy cope-b-a4b isvc in LW prod

https://gerrit.wikimedia.org/r/1304715

kevinbazira added a comment.Mon, Jun 22, 8:21 AM
In T427497#12033330, @isarantopoulos wrote:

Awesome work Kevin!
It would be helpful to run load tests against the v1/chat/completions endpoint and use a prompt of 1k tokens which would give us a better understanding of real world latencies. Unless @kostajh you are already doing this on your side, in which case we can skip additional load testing for now. We plan to tackle load testing and benchmarking of LLMs as a separate workstream.

We ran these load tests using the PSI team's cope policies. The cope-b-a4b isvc doesn't support the v1/chat/completions endpoint as that would require a request/response schema that's different from what we've set up with the PSI team above.

kevinbazira added a comment.Mon, Jun 22, 8:22 AM

Following T427497#12033298, the cope-b-a4b isvc has been deployed in LiftWing production (eqiad). It can be accessed by tools that run within the WMF production network, as shown below:

$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/cope-b-a4b:predict" -X POST \
-d '{ "content": "CLICK HERE TO WIN $10000!!! Visit http://totallylegit.biz NOW!!!", "policy": "Content must not contain spam, phishing attempts, or deceptive links." }' \
-H  "Host: cope-b-a4b.llm.wikimedia.org" \
-H "Content-Type: application/json" --http1.1

{"violation":1,"p_violation":1.0,"p_safe":1.522997974471263e-8}

real	0m0.046s
user	0m0.014s
sys	0m0.000s

We have also updated the liftwing_client to use this prod endpoint and shared it with the PSI team.

isarantopoulos added a comment.Mon, Jun 22, 8:23 AM

Sounds great, thanks for clarifying!

kevinbazira closed this task as Resolved.Mon, Jun 22, 8:23 AM
kevinbazira updated the task description. (Show Details)
kevinbazira moved this task from In Progress to Done on the Machine-Learning-Team (Q1 FY2026-27) board.
Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 22, 8:31 AM
kevinbazira mentioned this in T418267: Q2 FY2025-26 Goal: Host a content policy evaluation model on LiftWing.Tue, Jun 30, 10:21 AM
RKemper mentioned this in T431017: Pods evicted on ml-serve-eqiad due to disk pressure.Fri, Jul 3, 12:07 AM
isarantopoulos moved this task from Q1 FY2026-27 to Task Archive on the Machine-Learning-Team board.Mon, Jul 6, 7:49 AM
isarantopoulos edited projects, added: Machine-Learning-Team; removed: Machine-Learning-Team (Q1 FY2026-27).
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits