Page MenuHomePhabricator
Create Task
Maniphest T337213

Update to KServe 0.11
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

The KServe 0.11 release is about to be published, we should try to upgrade to it in order to get its latest features (including a configurable access log format that we'll need for T333804.

Overall plan:

  • - Check the release changelog to familiarize with what changed.
  • - Upgrade the Docker images in the production-images repo to the new version. This includes the storage-initializer one, that is the image ran by one of the containers in the model server' pods (so we'll need to wait the prerequisite of the next point before proceeding).
  • - Rollout the new model servers.
  • - Once kserve==0.11 is published in Pypi, we can start upgrading all the model server Docker images to it. The changes are not as big as the last upgrade, so I'd expect to see some dependency change but not much.
  • - Upgrade the KServe's deployment-charts chart with the new yaml config.
  • - Rollout the new control plane in k8s.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.May 22 2023, 7:45 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 22 2023, 7:45 AM
elukey added a comment.May 22 2023, 7:46 AM

https://github.com/kserve/kserve/releases/tag/v0.11.0-rc0

klausman claimed this task.May 22 2023, 7:56 AM
klausman mentioned this in T327241: Move the kserve custom helm chart to the upstream one.May 24 2023, 9:10 AM
klausman moved this task from Unsorted to Blocked on the Machine-Learning-Team board.May 25 2023, 10:57 AM
elukey moved this task from Blocked to Backlog/SRE on the Machine-Learning-Team board.Sep 5 2023, 1:09 PM
elukey mentioned this in T346446: Upgrade revscoring Docker images to KServe 0.11.Sep 15 2023, 1:26 PM
achou added a subtask: T347549: Upgrade outlink docker images to KServe 0.11.Sep 28 2023, 8:38 AM
achou added a subtask: T347550: Upgrade Revert Risk Language-agnostic docker images to KServe 0.11.Sep 28 2023, 8:42 AM
achou added a subtask: T347551: Upgrade Revert Risk Multilingual docker images to KServe 0.11.2.Sep 28 2023, 8:45 AM
gerritbot added a comment.Oct 23 2023, 9:26 AM

Change 967451 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/docker-images/production-images@master] images: Update kserve/build to kserve v0.11.1

https://gerrit.wikimedia.org/r/967451

gerritbot added a project: Patch-For-Review.Oct 23 2023, 9:26 AM
elukey moved this task from Backlog/SRE to In Progress on the Machine-Learning-Team board.Oct 23 2023, 10:17 AM
gerritbot added a comment.Oct 23 2023, 2:26 PM

Change 967451 merged by Klausman:

[operations/docker-images/production-images@master] images: Update kserve/build to kserve v0.11.1

https://gerrit.wikimedia.org/r/967451

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2023, 2:31 PM
klausman updated the task description. (Show Details)Oct 24 2023, 10:14 AM
calbon set the point value for this task to 5.Nov 2 2023, 7:12 PM
calbon triaged this task as Medium priority.Nov 2 2023, 7:25 PM
klausman added a comment.Nov 20 2023, 3:38 PM

0.11.2 was released recently. I will update the images to that version before proceeding.

gerritbot added a comment.Nov 20 2023, 3:39 PM

Change 975839 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/docker-images/production-images@master] kserve: Update images to v0.11.2

https://gerrit.wikimedia.org/r/975839

gerritbot added a project: Patch-For-Review.Nov 20 2023, 3:39 PM
gerritbot added a comment.Nov 20 2023, 4:03 PM

Change 975839 abandoned by Klausman:

[operations/docker-images/production-images@master] kserve: Update images to v0.11.2

Reason:

Broken change due to git shenanigans

https://gerrit.wikimedia.org/r/975839

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2023, 4:10 PM
klausman added a comment.Nov 20 2023, 5:48 PM

Images have been built and published:

```# docker images
REPOSITORY                                                                       TAG                            IMAGE ID       CREATED          SIZE
docker-registry.discovery.wmnet/kserve-storage-initializer                       0.11.2-1                       caed698a1bb0   6 minutes ago    727MB
docker-registry.discovery.wmnet/kserve-storage-initializer                       latest                         caed698a1bb0   6 minutes ago    727MB
docker-registry.discovery.wmnet/kserve-agent                                     0.11.2-1                       47980b2c48db   7 minutes ago    139MB
docker-registry.discovery.wmnet/kserve-agent                                     latest                         47980b2c48db   7 minutes ago    139MB
docker-registry.discovery.wmnet/kserve-controller                                0.11.2-1                       0ba5149f28fe   7 minutes ago    137MB
docker-registry.discovery.wmnet/kserve-controller                                latest                         0ba5149f28fe   7 minutes ago    137MB
docker-registry.discovery.wmnet/kserve-build                                     0.11.2-1                       852289661bd6   8 minutes ago    2.58GB
docker-registry.discovery.wmnet/kserve-build                                     latest                         852289661bd6   8 minutes ago    2.58GB
isarantopoulos closed subtask T347550: Upgrade Revert Risk Language-agnostic docker images to KServe 0.11 as Resolved.Nov 27 2023, 2:50 PM
achou closed subtask T347549: Upgrade outlink docker images to KServe 0.11 as Resolved.Dec 19 2023, 4:44 PM
klausman moved this task from In Progress to Ready To Go on the Machine-Learning-Team board.Jan 16 2024, 3:39 PM
isarantopoulos closed subtask T347551: Upgrade Revert Risk Multilingual docker images to KServe 0.11.2 as Resolved.Feb 12 2024, 5:06 PM
elukey updated the task description. (Show Details)Feb 28 2024, 1:16 PM

@klausman I can work on this if you want, IIUC the control plane is still on 0.10. Nothing major but better to be consistent, lemme know.

gerritbot added a comment.Feb 28 2024, 1:42 PM

Change 1007330 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: upgrade to upstream 0.11.2

https://gerrit.wikimedia.org/r/1007330

gerritbot added a project: Patch-For-Review.Feb 28 2024, 1:42 PM
elukey claimed this task.Feb 28 2024, 2:33 PM

Taking over from Tobias to free some tasks from his queue since I am just getting back to work (need something technical to do :D)

elukey moved this task from Ready To Go to In Progress on the Machine-Learning-Team board.Feb 28 2024, 2:34 PM
gerritbot added a comment.Feb 28 2024, 3:58 PM

Change 1007330 merged by Elukey:

[operations/deployment-charts@master] kserve: upgrade to upstream 0.11.2

https://gerrit.wikimedia.org/r/1007330

gerritbot added a comment.Feb 28 2024, 4:10 PM

Change 1007391 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: bump Docker image default versions to 0.11.2

https://gerrit.wikimedia.org/r/1007391

gerritbot added a comment.Feb 28 2024, 4:14 PM

Change 1007391 merged by Elukey:

[operations/deployment-charts@master] kserve: bump Docker image default versions to 0.11.2

https://gerrit.wikimedia.org/r/1007391

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2024, 4:31 PM
gerritbot added a comment.Feb 28 2024, 5:27 PM

Change 1007400 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] kserve: upgrade all images to Bookworm

https://gerrit.wikimedia.org/r/1007400

gerritbot added a project: Patch-For-Review.Feb 28 2024, 5:27 PM
gerritbot added a comment.Feb 29 2024, 2:25 PM

Change 1007624 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: update default Docker images

https://gerrit.wikimedia.org/r/1007624

gerritbot added a comment.Mar 1 2024, 1:02 PM

Change 1007400 merged by Elukey:

[operations/docker-images/production-images@master] kserve: upgrade all images to Bookworm

https://gerrit.wikimedia.org/r/1007400

gerritbot added a comment.Mar 1 2024, 1:37 PM

Change 1007624 merged by Elukey:

[operations/deployment-charts@master] kserve: update default Docker images

https://gerrit.wikimedia.org/r/1007624

gerritbot added a comment.Mar 1 2024, 1:48 PM

Change 1007903 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: add missing comma to kserve yaml config

https://gerrit.wikimedia.org/r/1007903

gerritbot added a comment.Mar 1 2024, 1:51 PM

Change 1007903 merged by Elukey:

[operations/deployment-charts@master] kserve: add missing comma to kserve yaml config

https://gerrit.wikimedia.org/r/1007903

gerritbot added a comment.Mar 1 2024, 2:05 PM

Change 1007907 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] kserve: use numeric id for nobody

https://gerrit.wikimedia.org/r/1007907

gerritbot added a comment.Mar 1 2024, 2:27 PM

Change 1007907 merged by Elukey:

[operations/docker-images/production-images@master] kserve: use numeric id for nobody

https://gerrit.wikimedia.org/r/1007907

gerritbot added a comment.Mar 1 2024, 2:30 PM

Change 1007915 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: bump docker image version for the storage-initializer

https://gerrit.wikimedia.org/r/1007915

gerritbot added a comment.Mar 1 2024, 2:51 PM

Change 1007915 merged by Elukey:

[operations/deployment-charts@master] kserve: bump docker image version for the storage-initializer

https://gerrit.wikimedia.org/r/1007915

gerritbot added a comment.Mar 1 2024, 3:10 PM

Change 1007923 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] kserve: add wmf-certificates in the right place for storage-init

https://gerrit.wikimedia.org/r/1007923

gerritbot added a comment.Mar 1 2024, 3:26 PM

Change 1007923 merged by Elukey:

[operations/docker-images/production-images@master] kserve: add wmf-certificates in the right place for storage-init

https://gerrit.wikimedia.org/r/1007923

Maintenance_bot removed a project: Patch-For-Review.Mar 1 2024, 3:31 PM
gerritbot added a comment.Mar 1 2024, 3:44 PM

Change rMW10079319463f had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: bump default Docker image for storage-init

https://gerrit.wikimedia.org/r/1007931

gerritbot added a project: Patch-For-Review.Mar 1 2024, 3:44 PM
gerritbot added a comment.Mar 1 2024, 3:48 PM

Change rMW10079319463f merged by Elukey:

[operations/deployment-charts@master] kserve: bump default Docker image for storage-init

https://gerrit.wikimedia.org/r/1007931

Maintenance_bot removed a project: Patch-For-Review.Mar 1 2024, 4:30 PM
elukey updated the task description. (Show Details)Mar 4 2024, 1:20 PM

The new control plane is running in staging, I took the opportunity to upgrade all Docker images to Debian Bookworm.

elukey added a comment.Mar 4 2024, 1:58 PM

Added https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/KServe#Admin_only_-_Upgrade_KServe_to_a_new_version to list the steps that we do now.

elukey added a comment.Mar 4 2024, 3:24 PM

Next steps: deploy to prod

elukey updated the task description. (Show Details)Mar 5 2024, 2:37 PM

New control plane deployed to ml-serve-{eqiad,codfw}, tested if killing a pod worked, all good.

elukey moved this task from In Progress to 2023-2024 Q3 Done on the Machine-Learning-Team board.Mar 5 2024, 2:38 PM
calbon closed this task as Resolved.Mar 5 2024, 3:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL