Page MenuHomePhabricator
Create Task
Maniphest T419722

Experiment with new kserve version on ml-staging-codfw
Closed, ResolvedPublic
Actions

Description

As per title, this will require staging env to be down for some time.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
fix: ml changelogsDpogorzelskioperations/docker-images/production-imagesmaster+7 -7
ml-serve: remove excludeIPRanges from cni configDpogorzelskioperations/deployment-chartsmaster+0 -11
kserve: update to version 0.17Dpogorzelskioperations/docker-images/production-imagesmaster+27 -3
ml-serve: add modified kserve 0.17 chartDpogorzelskioperations/deployment-chartsmaster+2 K -2
knative: update changelogsDpogorzelskioperations/docker-images/production-imagesmaster+42 -0
knative: update images to 1.21.1Dpogorzelskioperations/docker-images/production-imagesmaster+19 -3 K
kserve: bump to upstream version 0.17Dpogorzelskioperations/docker-images/production-imagesmaster+16 -0
kserve: bump to upstream version 0.17Dpogorzelskioperations/docker-images/production-imagesmaster+16 -0
Customize query in gerrit

Related Objects

Event Timeline

DPogorzelski-WMF created this task.Mar 11 2026, 3:26 PM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptMar 11 2026, 3:26 PM
gerritbot added a comment.Mar 16 2026, 1:44 PM

Change #1253495 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/docker-images/production-images@master] kserve: bump to upstream version 0.17

https://gerrit.wikimedia.org/r/1253495

gerritbot added a project: Patch-For-Review.Mar 16 2026, 1:44 PM

Change #1253495 abandoned by Dpogorzelski:

[operations/docker-images/production-images@master] kserve: bump to upstream version 0.17

https://gerrit.wikimedia.org/r/1253495

gerritbot added a comment.Mar 16 2026, 1:47 PM

Change #1253496 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/docker-images/production-images@master] kserve: bump to upstream version 0.17

https://gerrit.wikimedia.org/r/1253496

gerritbot added a comment.Mar 16 2026, 1:47 PM

Change #1253496 abandoned by Dpogorzelski:

[operations/docker-images/production-images@master] kserve: bump to upstream version 0.17

https://gerrit.wikimedia.org/r/1253496

gerritbot added a comment.Mar 16 2026, 1:51 PM

Change #1253498 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/docker-images/production-images@master] kserve: update to version 0.17

https://gerrit.wikimedia.org/r/1253498

gerritbot added a comment.Mar 16 2026, 1:52 PM

Change #1253498 abandoned by Dpogorzelski:

[operations/docker-images/production-images@master] kserve: update to version 0.17

https://gerrit.wikimedia.org/r/1253498

gerritbot added a comment.Mar 16 2026, 1:55 PM

Change #1253498 restored by Dpogorzelski:

[operations/docker-images/production-images@master] kserve: update to version 0.17

https://gerrit.wikimedia.org/r/1253498

gerritbot added a comment.Mar 16 2026, 3:14 PM

Change #1253498 merged by Dpogorzelski:

[operations/docker-images/production-images@master] kserve: update to version 0.17

https://gerrit.wikimedia.org/r/1253498

elukey subscribed.Mar 16 2026, 4:51 PM
isarantopoulos moved this task from Unsorted to In Progress on the Machine-Learning-Team board.Mar 24 2026, 2:26 PM
gerritbot added a comment.Mar 24 2026, 4:45 PM

Change #1260031 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/docker-images/production-images@master] knative: update images to 1.21.1

https://gerrit.wikimedia.org/r/1260031

gerritbot added a comment.Mar 25 2026, 10:31 AM

Change #1260031 merged by Dpogorzelski:

[operations/docker-images/production-images@master] knative: update images to 1.21.1

https://gerrit.wikimedia.org/r/1260031

gerritbot added a comment.Mar 25 2026, 11:36 AM

Change #1260656 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/docker-images/production-images@master] knative: update changelogs

https://gerrit.wikimedia.org/r/1260656

gerritbot added a comment.Mar 25 2026, 11:59 AM

Change #1260656 merged by Dpogorzelski:

[operations/docker-images/production-images@master] knative: update changelogs

https://gerrit.wikimedia.org/r/1260656

DPogorzelski-WMF added a comment.Mar 26 2026, 8:33 AM

Latest knative supported by kserve 0.17 seems to require a more recent kubernetes version:

{"severity":"EMERGENCY","timestamp":"2026-03-25T15:53:12.556721777Z","logger":"net-istio-controller","caller":"sharedmain/main.go:463","message":"Version check failed","commit":"1dc9b2d-dirty","knative.dev/pod":"net-istio-controller-6cc6d48947-v2t74","error":"kubernetes version \"1.31.4\" is not compatible, need at least \"1.33.0-0\" (this can be overridden with the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"knative.dev/pkg/injection/sharedmain.CheckK8sClientMinimumVersionOrDie\n\t/go/github.com/knative/net-istio/vendor/knative.dev/pkg/injection/sharedmain/main.go:463\nknative.dev/pkg/injection/sharedmain.MainWithConfig\n\t/go/github.com/knative/net-istio/vendor/knative.dev/pkg/injection/sharedmain/main.go:271\nknative.dev/pkg/injection/sharedmain.MainWithContext\n\t/go/github.com/knative/net-istio/vendor/knative.dev/pkg/injection/sharedmain/main.go:226\nmain.main\n\t/go/github.com/knative/net-istio/cmd/controller/main.go:31\nruntime.main\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.0.linux-amd64/src/runtime/proc.go:283"}

While forcing the run on an older k8s version might be possible I think I will stop here and keep the existing version of knative for now to unblock staging.
Currently it's hard to strike a balance between:

  • enforced k8s version
  • up to date external components
  • requirements for internal customizations
isarantopoulos mentioned this in T380722: Update kserve to v0.15.2* on ML clusters.Mar 26 2026, 8:38 AM
isarantopoulos mentioned this in T380723: Update knative-serving+net-istio to v1.12.x on ML clusters.
elukey added a comment.Mar 26 2026, 10:37 AM

@DPogorzelski-WMF my understanding from reading upstream's commits is that they bump the k8s version every now and then and they set it to the version that it was tested in their testing env:

https://github.com/knative-extensions/net-istio/commit/9af3f67f778c430e8b2f20a602a910c45f99c05c

It is probably very tough to keep up with their pace, and I doubt they are using anything that our current kube api can't support (new features etc..). I would personally try to force the KUBERNETES_MIN_VERSION for the net-istio plugin, they added it for this specific use case imho (namely folks with a slightly older kube environment).

DPogorzelski-WMF added a comment.Mar 26 2026, 1:33 PM

I will skip that for now as it's getting more complex than i initially anticipated. all services on staging work in the current setup and i'll ship kserve first and then circle back to this part

gerritbot added a comment.Mar 26 2026, 2:10 PM

Change #1261460 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/deployment-charts@master] ml-serve: add modified kserve 0.17 chart

https://gerrit.wikimedia.org/r/1261460

elukey added a comment.Mar 27 2026, 7:59 AM

@DPogorzelski-WMF as you prefer, but the difference between istio/kserve and knative is not really huge and I would personally upgrade first rather than keeping the old version. Also remember to check the new version of the knative chart too.

elukey renamed this task from Experiment with new kserve version on stagin to Experiment with new kserve version on ml-staging-codfw.Mar 27 2026, 8:51 AM
gerritbot added a comment.Apr 7 2026, 8:05 AM

Change #1261460 merged by jenkins-bot:

[operations/deployment-charts@master] ml-serve: add modified kserve 0.17 chart

https://gerrit.wikimedia.org/r/1261460

isarantopoulos edited projects, added: Machine-Learning-Team (Q1 FY2026-27); removed: Machine-Learning-Team.Apr 7 2026, 12:23 PM
isarantopoulos moved this task from Unsorted/Needs Triage to In Progress on the Machine-Learning-Team (Q1 FY2026-27) board.Apr 7 2026, 12:27 PM
DPogorzelski-WMF closed this task as Resolved.Apr 7 2026, 3:02 PM
DPogorzelski-WMF claimed this task.
isarantopoulos mentioned this in T422591: Update kserve Python package to 0.17 across all inference services.Apr 8 2026, 8:47 AM
elukey reopened this task as Open.EditedApr 8 2026, 9:06 AM

@DPogorzelski-WMF there is another thing to test imho before proceeding with prod, namely https://github.com/kserve/kserve/pull/3316

We currently need to add the following bits to the kserve pods to allow the storage initializer to skip the istio's sidecar traffic policies when running:

traffic.sidecar.istio.io/excludeOutboundIPRanges: 10.2.2.54/32,10.2.1.54/32

The above are the hardcoded IPs of thanos-swift.discovery.wmnet. IIUC with the aforementioned patch we don't need them anymore, since the storage initializer's uid is automatically set. In theory, the change should just be removing:

custom_deploy.d/istio/ml-serve/config_1.24.2.yaml:        excludeIPRanges: "10.2.2.54/32,10.2.1.54/32"

Then deploy it via istioctl and restart pods in the various namespaces to see if it works.

DPogorzelski-WMF added a comment.Apr 8 2026, 1:27 PM

cool, i'll try

DPogorzelski-WMF added a comment.Apr 20 2026, 9:22 AM

tested on edit-check, seems to be working fine

gerritbot added a comment.Apr 20 2026, 9:27 AM

Change #1275354 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/deployment-charts@master] ml-serve: remove excludeIPRanges from cni config

https://gerrit.wikimedia.org/r/1275354

gerritbot added a comment.Apr 27 2026, 12:09 PM

Change #1275354 merged by jenkins-bot:

[operations/deployment-charts@master] ml-serve: remove excludeIPRanges from cni config

https://gerrit.wikimedia.org/r/1275354

isarantopoulos subscribed.May 12 2026, 1:08 PM

everything in staging seems to be going fine and no issue has been reported. We have decided to move forward and roll these changes to production next week Monday 18/5
Staging versions:
kserve: 0.17
knative: v1.21.1

DPogorzelski-WMF closed this task as Resolved.May 20 2026, 8:37 AM
DPogorzelski-WMF moved this task from Q1 FY2026-27 to 2025-2026 Q2 Done on the Machine-Learning-Team board.
DPogorzelski-WMF edited projects, added: Machine-Learning-Team; removed: Machine-Learning-Team (Q1 FY2026-27).
isarantopoulos moved this task from 2025-2026 Q2 Done to Q1 FY2026-27 on the Machine-Learning-Team board.May 20 2026, 8:42 AM
isarantopoulos edited projects, added: Machine-Learning-Team (Q1 FY2026-27); removed: Machine-Learning-Team.
isarantopoulos moved this task from In Progress to Done on the Machine-Learning-Team (Q1 FY2026-27) board.
JMeybohm reopened this task as Open.May 27 2026, 8:57 AM
JMeybohm subscribed.

@DPogorzelski-WMF all these[1] changelog entries have wrongly formatted footers (maintainer name and email, see https://manpages.debian.org/testing/dpkg-dev/deb-changelog.5.en.html) which leads to multiple warnings from docker-pkg. Would you mind fixing please?

[1] https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1260656

DPogorzelski-WMF added a comment.May 27 2026, 9:07 AM

sure thing

gerritbot added a comment.May 27 2026, 12:22 PM

Change #1294268 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/docker-images/production-images@master] fix: ml changelogs

https://gerrit.wikimedia.org/r/1294268

gerritbot added a comment.May 27 2026, 1:14 PM

Change #1294268 merged by Dpogorzelski:

[operations/docker-images/production-images@master] fix: ml changelogs

https://gerrit.wikimedia.org/r/1294268

DPogorzelski-WMF closed this task as Resolved.Tue, Jun 23, 1:10 PM
isarantopoulos moved this task from Q1 FY2026-27 to Task Archive on the Machine-Learning-Team board.Mon, Jul 6, 7:49 AM
isarantopoulos edited projects, added: Machine-Learning-Team; removed: Machine-Learning-Team (Q1 FY2026-27).
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits