Page MenuHomePhabricator
Create Task
Maniphest T311982

Upgrade ml clusters to kserve 0.8
Closed, ResolvedPublic
Actions

Description

kserve 0.8 release is out: https://github.com/kserve/kserve/releases/tag/v0.8.0

There are some interesting new things, we should try to upgrade and see if everything works as expected in staging.

Some notes:

  • knative recommended version is now 1.0, meanwhile we have 0.18. In https://github.com/kserve/kserve/issues/2292 upstream suggested that everything should work fine even on our platform, but we'd need to test.
  • As explained in the changelog, several Python classes got changed from 0.7 to 0.8, so we'd need to change our code as well.

High level list of things to do:

  1. Prepare the new docker image in production-images (kserve controller).
  2. File code changes for inference-service model.py files and create the new Docker images (bump kserve's pypi dependency as well).
  3. Import the new kserve yaml config in deployment-charts and update the related chart.
  4. Deploy everything to staging and test.
  5. Deploy everything to prod and test.

Details

SubjectRepoBranchLines +/-
ml-services: move prod docker images to KServe 0.8operations/deployment-chartsmaster+6 -30
kserve: update storage-initializer's docker image versionoperations/deployment-chartsmaster+2 -2
kserve: add service account to StatefulSetoperations/deployment-chartsmaster+2 -1
kserve: apply upstream fix to storage-initializeroperations/docker-images/production-imagesmaster+16 -7
kserve: upgrade to upstream release 0.8operations/deployment-chartsmaster+2 K -44
ml-services: update Docker images to KServe 0.8 in stagingoperations/deployment-chartsmaster+24 -3
ml-services: test the first kserve 0.8 Docker imagesoperations/deployment-chartsmaster+3 -1
Update Python model servers and requirements to KServe 0.8machinelearning/liftwing/inference-servicesmain+62 -62
Upgrade kserve images to upstream release 0.8operations/docker-images/production-imagesmaster+28 -4
Customize query in gerrit

Related Objects

Event Timeline

elukey created this task.Jul 4 2022, 8:17 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 4 2022, 8:17 AM
gerritbot added a comment.Jul 4 2022, 8:53 AM

Change 810841 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] Upgrade kserve images to upstream release 0.8

https://gerrit.wikimedia.org/r/810841

gerritbot added a project: Patch-For-Review.Jul 4 2022, 8:53 AM
elukey moved this task from Unsorted to Active Tasks on the Machine-Learning-Team board.Jul 5 2022, 2:10 PM
elukey edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.
achou subscribed.Jul 7 2022, 1:42 PM
gerritbot added a comment.Jul 7 2022, 3:21 PM

Change 810841 merged by Elukey:

[operations/docker-images/production-images@master] Upgrade kserve images to upstream release 0.8

https://gerrit.wikimedia.org/r/810841

Maintenance_bot removed a project: Patch-For-Review.Jul 7 2022, 3:30 PM
elukey added a comment.Jul 7 2022, 3:31 PM
root@build2001:/srv/images/production-images# build-production-images 
== Step 0: scanning /srv/images/production-images/images ==
Will build the following images:
* docker-registry.discovery.wmnet/kserve-build:0.8.0-1
* docker-registry.discovery.wmnet/kserve-controller:0.8.0-1
* docker-registry.discovery.wmnet/kserve-agent:0.8.0-1
* docker-registry.discovery.wmnet/kserve-storage-initializer:0.8.0-1
== Step 1: building images ==
* Built image docker-registry.discovery.wmnet/kserve-build:0.8.0-1
* Built image docker-registry.discovery.wmnet/kserve-controller:0.8.0-1
* Built image docker-registry.discovery.wmnet/kserve-agent:0.8.0-1
* Built image docker-registry.discovery.wmnet/kserve-storage-initializer:0.8.0-1
== Step 2: publishing ==
Successfully published image docker-registry.discovery.wmnet/kserve-controller:0.8.0-1
Successfully published image docker-registry.discovery.wmnet/kserve-agent:0.8.0-1
Successfully published image docker-registry.discovery.wmnet/kserve-build:0.8.0-1
Successfully published image docker-registry.discovery.wmnet/kserve-storage-initializer:0.8.0-1
== Build done! ==
You can see the logs at ./docker-pkg-build.log
== Step 0: scanning /srv/images/production-images/istio ==
Will build the following images:
== Step 1: building images ==
== Step 2: publishing ==
== Build done! ==
You can see the logs at ./docker-pkg-build.log
== Step 0: scanning /srv/images/production-images/cert-manager ==
Will build the following images:
== Step 1: building images ==
== Step 2: publishing ==
== Build done! ==
You can see the logs at ./docker-pkg-build.log
elukey claimed this task.Jul 19 2022, 2:29 PM
elukey moved this task from Parked to In Progress on the Machine-Learning-Team (Active Tasks) board.
achou added a comment.Jul 19 2022, 3:42 PM

I ran into a dependency issue when building docker image after bumping kserve’s dependencies in requirements.txt:

  • kserve==0.8.0
  • ray==1.9.0
  • numpy==1.19.2 (kserve 0.8.0 depends on numpy~=1.19.2)
  • boto3==1.20.24 (kserve 0.8.0 depends on boto3==1.20.24)
  • botocore==1.23.24 (boto3 1.20.24 depends on botocore<1.24.0 and >=1.23.24)
#18 174.4 ERROR: Cannot install -r model-server/requirements.txt (line 103), -r model-server/requirements.txt (line 108), -r model-server/requirements.txt (line 40), -r model-server/requirements.txt (line 62), -r model-server/requirements.txt (line 63) and numpy==1.19.2 because these package versions have conflicting dependencies.
#18 174.4 
#18 174.4 The conflict is caused by:
#18 174.4     The user requested numpy==1.19.2
#18 174.4     gensim 3.8.3 depends on numpy>=1.11.3
#18 174.4     konlpy 0.5.2 depends on numpy>=1.6
#18 174.4     kserve 0.8.0 depends on numpy~=1.19.2
#18 174.4     ray 1.9.0 depends on numpy>=1.16; python_version < "3.9"
#18 174.4     revscoring 2.11.4 depends on numpy<1.18.999 and >=1.18.4
#18 174.4 
#18 174.4 To fix this you could try to:
#18 174.4 1. loosen the range of package versions you've specified
#18 174.4 2. remove package versions to allow pip attempt to solve the dependency conflict
#18 174.4 
#18 174.4 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
------
executor failed running [/bin/sh -c python3.7 "-m" "pip" "wheel" "-r" "model-server/requirements.txt" && python3.7 "-m" "pip" "install" "--target" "/opt/lib/python/site-packages" "-r" "model-server/requirements.txt"]: exit code: 1
aikochou@Aikos-MacBook-Pro editquality %

We need to check if we can bump revscoring's dependencies numpy to 1.19.2

elukey added a comment.EditedJul 20 2022, 8:06 AM

@achou thanks a lot! I have tested revscoring on stat1004 in the following way:

  • cloned the revscoring repo
  • pip installed dependencies, and update numpy to 1.19.2
  • ran my script for T309623#8084829, including getting a score from the model.

Everything worked nicely, so I think we can safely upgrade the requirements.txt file in the revscoring repo. Since we have some things pending (MWAPICache, a community member fixing tests, numpy upgrade, etc..) I think that we can wait for everything to be merged and then we can cut a new release (probably 2.11.5).

Ran also all unit tests, only 3 are failing but it is not a regression (namely they are already failing now).

gerritbot added a comment.Jul 20 2022, 9:10 AM

Change 815691 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: upgrade to upstream release 0.8

https://gerrit.wikimedia.org/r/815691

gerritbot added a project: Patch-For-Review.Jul 20 2022, 9:10 AM
gerritbot added a comment.Jul 20 2022, 1:49 PM

Change 815721 had a related patch set uploaded (by Elukey; author: Elukey):

[machinelearning/liftwing/inference-services@main] Update Python model servers and requirements to KServe 0.8

https://gerrit.wikimedia.org/r/815721

gerritbot added a comment.Jul 25 2022, 9:00 AM

Change 815721 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] Update Python model servers and requirements to KServe 0.8

https://gerrit.wikimedia.org/r/815721

elukey mentioned this in rMLIS7fd064807810: Update Python model servers and requirements to KServe 0.8.Jul 25 2022, 9:02 AM
gerritbot added a comment.Jul 25 2022, 9:19 AM

Change 816710 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] ml-services: test the first kserve 0.8 Docker images

https://gerrit.wikimedia.org/r/816710

gerritbot added a comment.Jul 25 2022, 9:24 AM

Change 816710 merged by Elukey:

[operations/deployment-charts@master] ml-services: test the first kserve 0.8 Docker images

https://gerrit.wikimedia.org/r/816710

elukey added a comment.Jul 25 2022, 9:37 AM

Tested all Docker images locally (and added documentation on Wikitech). Merged the change and updated the isvc images for articlequality and editquality in staging, all tests passed.

It seems possible, at least between these two kserve versions, to upgrade the Docker images separately from the Kserve control plane.

gerritbot added a comment.Jul 25 2022, 9:46 AM

Change 816716 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] ml-services: update Docker images to KServe 0.8 in staging

https://gerrit.wikimedia.org/r/816716

gerritbot added a comment.Jul 25 2022, 10:19 AM

Change 816716 merged by Elukey:

[operations/deployment-charts@master] ml-services: update Docker images to KServe 0.8 in staging

https://gerrit.wikimedia.org/r/816716

gerritbot added a comment.Jul 25 2022, 12:58 PM

Change 815691 merged by Elukey:

[operations/deployment-charts@master] kserve: upgrade to upstream release 0.8

https://gerrit.wikimedia.org/r/815691

gerritbot added a comment.Jul 25 2022, 1:35 PM

Change 816762 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: add service account to StatefulSet

https://gerrit.wikimedia.org/r/816762

gerritbot added a comment.Jul 25 2022, 2:35 PM

Change 816792 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] kserve: apply upstream fix to storage-initializer

https://gerrit.wikimedia.org/r/816792

gerritbot added a comment.Jul 25 2022, 3:29 PM

Change 816792 merged by Elukey:

[operations/docker-images/production-images@master] kserve: apply upstream fix to storage-initializer

https://gerrit.wikimedia.org/r/816792

gerritbot added a comment.Jul 25 2022, 3:34 PM

Change 816807 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: update storage-initializer's docker image version

https://gerrit.wikimedia.org/r/816807

gerritbot added a comment.Jul 25 2022, 3:35 PM

Change 816762 merged by Elukey:

[operations/deployment-charts@master] kserve: add service account to StatefulSet

https://gerrit.wikimedia.org/r/816762

gerritbot added a comment.Jul 25 2022, 4:17 PM

Change 816807 merged by Elukey:

[operations/deployment-charts@master] kserve: update storage-initializer's docker image version

https://gerrit.wikimedia.org/r/816807

elukey added a comment.Jul 26 2022, 8:49 AM

The new storage-initializer image works! KServe 0.8 is deployed in staging and so far everything works fine. The next step is to plan and execute the deployment to production.

High level details:

  • Move all Kserve pods to the new Docker images.
  • Upgrade the K8s control plane to 0.8
gerritbot added a comment.Jul 27 2022, 9:24 AM

Change 817732 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] ml-services: move prod docker images to KServe 0.8

https://gerrit.wikimedia.org/r/817732

gerritbot added a comment.Jul 27 2022, 1:21 PM

Change 817732 merged by Elukey:

[operations/deployment-charts@master] ml-services: move prod docker images to KServe 0.8

https://gerrit.wikimedia.org/r/817732

elukey added a comment.Jul 27 2022, 4:13 PM

ml-serve-codfw upgraded, all good up to now, waiting a day and some deployments before proceeding wit eqiad as well.

elukey closed this task as Resolved.Jul 28 2022, 1:15 PM

ml-serve-eqiad completed as well.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL