Update knative's queue proxy image and the Swift/S3 accounts used on ml-serve clusters
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	elukey
	Jul 3 2025, 7:07 AM

Project Tags

Referenced Files

None

Subscribers

Description

We found two issues in the current ml-serve setup:

the Knative's queue proxy image (deployed on all isvc pods) is still running Debian Buster (now EOL), our last upgrade didn't pick up the Bookworm-based image. https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1165850 should be deployed to all clusters, and new images should pick up the new queue proxy during the next deployments.
the mlserve:prod (read-write account) is being used instead of mlserve:ro, so the storage initializer on ml-serve-codfw and ml-serve-eqiad is using the wrong credentials. Staging is fine, so it is a proof that it works fine. We should apply the correct account to all isvcs in production to add extra security (namely, any compromised pod will not be able to modify anything on S3/Swift).

The former patch requires an SRE to merge and deploy it since it is related to admin_ng (it sets the image at the Knative level), but I am not 100% sure if after that an helmfile diff would show anything to deploy for a given isvc. Needs to be tested in staging first.

The latter requires an SRE to make a change in puppet private, and after that helmfile diff should show the right change for all the isvc deployments.

We should couple both these changes and roll them out together to limit the amount of deployments needed.

Details

Other Assignee: achou

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T398948 Q1 FY2025-26 Goal: Operational Excellence - LiftWing Platform Updates & Improvements
		Resolved		kevinbazira	T398533 Update knative's queue proxy image and the Swift/S3 accounts used on ml-serve clusters

Event Timeline

elukey created this task.Jul 3 2025, 7:07 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2025, 7:07 AM

I'll be working on this for MLOps Week. @klausman Could you assist with this? I plan to test in staging first.

For the first issue, we already have the patch ready. I have a few questions to better understand the logic: Why don't we also change the image version in the chart here? When we define the version in the admin_ng, does it overwrite the chart setting? Why didn't we have it in the admin_ng before?

After deployment, we can use kubectl describe pod to check what image version the pod uses, as follows:

queue-proxy:
  Container ID:   containerd://6f00ef7e49c98da65350470f709113c8e24cf61731444df9558b03f4f959dab3
  Image:          docker-registry.discovery.wmnet/knative-serving-queue:1.7.2-1
  Image ID:       docker-registry.discovery.wmnet/knative-serving-queue@sha256:fe15e1d06f22cb7092114132d39f764e7e9a2cd096582fbaafd589a319d73259
  Ports:          8022/TCP, 9090/TCP, 9091/TCP, 8012/TCP, 8112/TCP
  Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
  State:          Running
    Started:      Thu, 03 Jul 2025 12:33:36 +0000
  Ready:          True

For the second issue, it requires making a change in puppet private. IIUC, we already have two accounts mlserve:prod and mlserve:ro defined in thanos/swift.yaml, but where do we config the account used by the storage initializer?

In T398533#10983785, @achou wrote:

@klausman Could you assist with this? I plan to test in staging first.

Of course!

For the second item, as I understand it, staging already uses mlserve_ro:

(from hieradata/role/common/deployment_server/kubernetes.yaml in the private repo:

profile::kubernetes::deployment_server_secrets::services:
  ml-serve:
    revscoring-editquality-goodfaith:
      ml-staging-codfw:
        secrets: &kserve-swift-secret-staging
          - name: 'swift-s3-credentials'
            type: Opaque
            annotations:
              serving.kserve.io/s3-endpoint: thanos-swift.discovery.wmnet
              serving.kserve.io/s3-region: us-east-1
              serving.kserve.io/s3-usehttps: "1"
              serving.kserve.io/s3-cabundle: "/etc/ssl/certs/wmf-ca-certificates.crt"
            data:
              AWS_ACCESS_KEY_ID: mlserve:ro
              AWS_SECRET_ACCESS_KEY: [elided]

The other services then just reference that bit using the *kserve-swift-secret reference.

So we'd change the currently-different sections in that file to use uniform secrets, I can take care of that. I will ping you on IRC once it's done.

The first item (knative image update), we can proceed whenever we both have a time window to keep an eye on things.

isarantopoulos added a project: Essential-Work.Jul 8 2025, 1:56 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-10T08:05:52Z] <klausman> Depooling Liftwing prod in codfw so we can roll out some changes that restart all services (cf. T398533)

isarantopoulos added a parent task: T398948: Q1 FY2025-26 Goal: Operational Excellence - LiftWing Platform Updates & Improvements.Jul 11 2025, 9:51 AM

Changes have been deployed in codfw (ml-staging-codfw and ml-serve-codfw). eqiad will need to be depooled and have the same changes applied (knative + credential change).

As we prepare to deploy in eqiad, at the moment, these 3 httpbb tests fail:

kevinbazira@deploy1003:~$ httpbb /srv/deployment/httpbb-tests/liftwing/production/test_editcheck.yaml --hosts inference.svc.eqiad.wmnet --https_port 30443
Sending to inference.svc.eqiad.wmnet...
https://edit-check.edit-check.wikimedia.org/v1/models/edit-check:predict (/srv/deployment/httpbb-tests/liftwing/production/test_editcheck.yaml:3)
    Body: expected to contain 'probability', got '{"predictions":[{"status_code":400,"errors":["Erro'... (94 characters total).
===
FAIL: 1 request sent to inference.svc.eqiad.wmnet. 1 request with failed assertions.
kevinbazira@deploy1003:~$ 
kevinbazira@deploy1003:~$ 
kevinbazira@deploy1003:~$ httpbb /srv/deployment/httpbb-tests/liftwing/production/test_recommendation-api-ng.yaml --hosts inference.svc.eqiad.wmnet --https_port 30443
Sending to inference.svc.eqiad.wmnet...
https://recommendation-api-ng.discovery.wmnet:31443/service/lw/recommendation/api/v1/translation (/srv/deployment/httpbb-tests/liftwing/production/test_recommendation-api-ng.yaml:3)
    ERROR: HTTPSConnectionPool(host='inference.svc.eqiad.wmnet', port=30443): Max retries exceeded with url: /service/lw/recommendation/api/v1/translation (Caused by SSLError(SSLCertVerificationError("hostname 'recommendation-api-ng.discovery.wmnet:31443' doesn't match either of 'inference.discovery.wmnet', 'inference.svc.codfw.wmnet', 'inference.svc.eqiad.wmnet', '*.revscoring-editquality-goodfaith.wikimedia.org', '*.revscoring-editquality-damaging.wikimedia.org', '*.revscoring-editquality-reverted.wikimedia.org', '*.revscoring-draftquality.wikimedia.org', '*.revscoring-drafttopic.wikimedia.org', '*.revscoring-articlequality.wikimedia.org', '*.revscoring-articletopic.wikimedia.org', '*.experimental.wikimedia.org', '*.articletopic-outlink.wikimedia.org', '*.article-descriptions.wikimedia.org', '*.revertrisk.wikimedia.org', '*.llm.wikimedia.org', '*.logo-detection.wikimedia.org', '*.article-models.wikimedia.org', '*.revision-models.wikimedia.org', '*.edit-check.wikimedia.org'")))
===
ERRORS: 1 request attempted to inference.svc.eqiad.wmnet. Errors connecting to 1 host.
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$
kevinbazira@deploy1003:~$ httpbb /srv/deployment/httpbb-tests/liftwing/production/test_article-descriptions.yaml --hosts inference.svc.eqiad.wmnet --https_port 30443
Sending to inference.svc.eqiad.wmnet...
https://article-descriptions.article-descriptions.wikimedia.org/v1/models/article-descriptions:predict (/srv/deployment/httpbb-tests/liftwing/production/test_article-descriptions.yaml:3)
    Status code: expected 200, got 500.
    Body: expected to contain 'prediction', got '{"error":"KeyError : None"}'.
===
FAIL: 1 request sent to inference.svc.eqiad.wmnet. 1 request with failed assertions.

UPDATE: Third test now passes. It probably was an intermittent issue:

$ httpbb /srv/deployment/httpbb-tests/liftwing/production/test_article-descriptions.yaml --hosts inference.svc.eqiad.wmnet --https_port 30443
Sending to inference.svc.eqiad.wmnet...

The edit check failure is likely due to the recent change of schema that now requires the page_title (so in the above request you end up getting a 400)

The rec-api-ng failures I think are related to what Aiko mentioned yesterday -- this is not an inference service requests are made to the discovery endpoint instead of the specific cluster but I'm not sure about the error above.
cc: @achou

@kevinbazira The first issue (edit-check) is occurring because the patch (updating edit-check httpbb tests for page_title https://gerrit.wikimedia.org/r/c/operations/puppet/+/1167858) hasn't been merged. I'll merge it later.

For the second issue (rec-api-ng), it's the one I mentioned in the MLOps Week Log.
Since rec-api-ng doesn't use the inference endpoint (inference.svc.xxx.wmnet), we need to figure out the correct endpoint to use.

Ah, I can't merge it. Only SREs have +2 for puppet changes.

Thanks to @elukey, who provided support on this task, we have deployed the new knative queue proxy image and Swift/S3 credentials in eqiad.

isarantopoulos assigned this task to kevinbazira.Jul 15 2025, 1:27 PM

isarantopoulos moved this task from Unsorted to In Progress on the Machine-Learning-Team board.

isarantopoulos updated Other Assignee, added: achou.

isarantopoulos closed this task as Resolved.Jul 15 2025, 2:51 PM

isarantopoulos moved this task from In Progress to 2025-2026 Q1 Done on the Machine-Learning-Team board.

Update knative's queue proxy image and the Swift/S3 accounts used on ml-serve clustersClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Update knative's queue proxy image and the Swift/S3 accounts used on ml-serve clusters
Closed, ResolvedPublic
Actions

Related Objects
Search...