We need to install KFServing standalone our ml-serve k8s cluster.
Requirements:
Install docs: https://github.com/kubeflow/kfserving#standalone-kfserving-installation
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T272917 Lift Wing proof of concept | |||
| Resolved | elukey | T272919 Install KFServing standalone | |||
| Resolved | elukey | T278194 Install Knative on ml-serve cluster | |||
| Resolved | elukey | T278192 Install Istio on ml-serve cluster | |||
| Declined | None | T280661 Install certmanager on ml-serve cluster (if needed) |
Here are my notes from installing KFServing on GKE last quarter: https://etherpad.wikimedia.org/p/kfserving-standalone
TLDR: there is a weird version mismatch issue between Knative & Istio that can potentially prevent us from using the transformer (pre/post-processing) and explainabilty features in KFServing.
@calbon mentioned in chat:
KFServing's explainability feature (i.e. an API endpoint for each model that provides some insights into how a model came to some conclusion) is a nice-to-have. However, transformers (i.e. a feature that allows api requests to be pre-processed before being submitted to a model for a prediction) is critical because it is part of the current ORES featureset
For now let's try to just follow the directions in the KFServing README and setup Istio to handle cluster-interal traffic so we can use transformers.
I found an example of using helm to install the KFServing dependencies:
https://github.com/kubeflow/kfserving/blob/8a8f33a85a12ea4813f726b41581cbf9335ede9b/test/scripts/run-e2e-tests.sh#L57
you might also find the quick install script interesting, it has a more fresh version of Istio&knative (1.6 & 0.18) than the run-e2e-tests.sh (1.3 & 0.17). I'm currently rewriting the run-e2e-tests.sh to migrate the test to a tekton pipeline and will keep you posted. I'm targeting Istio 1.7 and Knative 0.20. One more thing, the helm installation method is nice, but both Istio&Knative also have operators for installation nowadays, which can make future upgrades easier.
Thanks @Theofpa ! We are considering helm as it is part of our SRE stack used across the Foundation, however I can see the operators being very beneficial for long term use.
@ACraze @kevinbazira I was reviewingthe kfserving.yaml kubernetes config looking for Docker images to build, and beside the kfserving ones (controller/agent) I found also Docker images (shipped to Dockerhub) based on: https://github.com/kubeflow/kfserving/tree/master/python
It seems mostly related to model servers for various providers, but I have no idea if we need them now or not. Can you shed some light? :D
Change 693644 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/production-images@master] Add base kubeflow kfserving images and kube-rbac-proxy
These are the current locations of the images published and/or used by kfserving:
| Registry | Component | Current Image location |
|---|---|---|
| Docker Hub | AIX360 | docker.io/kfserving/aix-explainer |
| Docker Hub | Alibi | docker.io/kfserving/alibi-explainer |
| Docker Hub | ART | docker.io/kfserving/art-explainer |
| Docker Hub | LightGBM | docker.io/kfserving/lgbserver |
| Docker Hub | PMML | docker.io/kfserving/pmmlserver |
| Docker Hub | TorchServe | docker.io/kfserving/torchserve-kfs |
| Docker Hub | Agent | docker.io/kfserving/agent |
| Docker Hub | tf-serving | docker.io/tensorflow/serving |
| Docker Hub | mlserver | docker.io/seldonio/mlserver |
| GCR | pytorch | gcr.io/kfserving/pytorchserver |
| GCR | sklearn | gcr.io/kfserving/sklearnserver |
| GCR | xgboost | gcr.io/kfserving/xgbserver |
| GCR | storage-init | gcr.io/kfserving/storage-initializer |
| GCR | controller | gcr.io/kfserving/kfserving-controller |
| GCR | rbac-proxy | gcr.io/kubebuilder/kube-rbac-proxy |
| MCR | ONNX | mcr.microsoft.com/onnxruntime/server |
| NVCR | triton | nvcr.io/nvidia/tritonserver |
To help you answer the question, these images are mostly model servers (tensorflow, pytorch, sklearn, etc) and explainers. The current models built by @kevinbazira and @ACraze are based on a custom image rather than one of the popular model servers. So if you need to have the minimum list of images cloned in your registry to make kfserving work for the current capabilities of the platform, as you said these would be the controller and the agent.
Although from the roadmap of lift-wing I understand that the platform will serve more model types in the future, which most likely will be using one of the popular frameworks like tensorflow, pytorch or sklearn. So my recommendation would be to cache them all.
One more thing, we're planning to move out from docker hub in the near future and have the images hosted in aws ecr-public.
It seems mostly related to model servers for various providers, but I have no idea if we need them now or not. Can you shed some light? :D
@elukey -- mostly echoing @Theofpa: I think all we need right now for the MVP is controller & agent from KFServing. The ORES models will be a custom image that we are still finishing and the other model we are working on is the Outlinks topic model, which is another custom image that runs a fastText model. We also might need to do the storage-init as well, but that depends on the outcome of T282802: Implement model storage for enwiki-goodfaith inference service
Long-term we will definitely want most (if not all) of the model servers. I know some teams are using tensorflow and pytorch for upcoming projects, also sklearn is pretty common too.
Change 693644 merged by Elukey:
[operations/docker-images/production-images@master] Add base kubeflow kfserving images
Change 700179 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/production-images@master] Move knative serving's queue image to a different layout
Change 700179 merged by Elukey:
[operations/docker-images/production-images@master] Move knative serving's queue image to a different layout
Change 700470 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] WIP - Add kubeflow's kfserving chart
We are currently try to deploy layer by layer in production, following this order:
Setting this task to STALLED until it will be actionable, see the other subtasks of the parent for more info.
Change 708783 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/production-images@master] Import Kubeflow Kfserving 0.6.0
Change 708783 merged by Elukey:
[operations/docker-images/production-images@master] Import Kubeflow Kfserving 0.6.0
Change 709014 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] Add ml-serve-{eqiad,codfw} to kubernetes_clusters
Change 709014 merged by Elukey:
[operations/puppet@production] Add ml-serve-{eqiad,codfw} to kubernetes_clusters
Change 709494 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] Add kfserving basic helmfile config under admin_ng
Change 700470 merged by Elukey:
[operations/deployment-charts@master] Add kubeflow's kfserving charts
Change 709494 merged by Elukey:
[operations/deployment-charts@master] Add kfserving basic helmfile config under admin_ng
Change 710226 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] Improve the kubeflow-kfserving chart
Change 710226 merged by Elukey:
[operations/deployment-charts@master] Improve the kubeflow-kfserving chart
Change 710296 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] helmfile: allow kubeflow-kfserving to create the kfserving-system ns
Change 710296 merged by Elukey:
[operations/deployment-charts@master] helmfile: allow kubeflow-kfserving to create the kfserving-system ns
Change 710301 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] Kubeflow: fix secrete in chart and update helmfile
Change 710301 merged by Elukey:
[operations/deployment-charts@master] Kubeflow: fix secret in chart and update helmfile
Change 710307 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow-kfserving: better handling of Secrets
Change 710307 merged by Elukey:
[operations/deployment-charts@master] kubeflow-kfserving: better handling of Secrets
Change 710481 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow: create a separate chart for its Secret
Change 710483 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow: add pre-hook ordering for Namespace and Secret
Change 710481 abandoned by Elukey:
[operations/deployment-charts@master] kubeflow: create a separate chart for its Secret
Reason:
Change 710483 merged by Elukey:
[operations/deployment-charts@master] kubeflow: add pre-hook ordering for Namespace and Secret
Change 710486 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow: fix the cert name for the webhook TLS certificate
Change 710486 merged by Elukey:
[operations/deployment-charts@master] kubeflow: fix the cert name for the webhook TLS certificate
Change 710493 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow: add container env variables to reach the k8s api
Change 710493 merged by Elukey:
[operations/deployment-charts@master] kubeflow: add container env variables to reach the k8s api
Finally!
elukey@ml-serve-ctrl1001:~$ sudo kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE istio-system cluster-local-gateway-585f96dccc-54kqm 1/1 Running 1 13d istio-system istio-ingressgateway-7ffffd874b-67zkp 1/1 Running 1 13d istio-system istiod-68d4cb6c9-h84qd 1/1 Running 1 13d kfserving-system kfserving-controller-manager-0 1/1 Running 0 7m59s knative-serving activator-867d54cc88-vwdpt 1/1 Running 2 8d knative-serving autoscaler-cfc4cc49f-zbvmv 1/1 Running 0 8d knative-serving controller-784f95f8df-m4djc 1/1 Running 0 8d knative-serving istio-webhook-b8854d86f-4lxh2 1/1 Running 0 8d knative-serving networking-istio-857f9bbdf6-bdcp5 1/1 Running 0 8d knative-serving webhook-5bf64fb48d-qvr8x 1/1 Running 0 8d
Change 710584 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/production-images@master] Add the Kubeflow storage initializer docker image
Change 710584 merged by Elukey:
[operations/docker-images/production-images@master] Add the Kubeflow storage initializer docker image
Change 711096 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow-kfserving: move to the Wikimedia storage-initializer
Change 711096 merged by Elukey:
[operations/deployment-charts@master] kubeflow-kfserving: move to the Wikimedia storage-initializer
Change 711113 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow,knative: use new controller image and docker registry endpoint
Change 711113 merged by Elukey:
[operations/deployment-charts@master] kubeflow,knative: use new controller image and docker registry endpoint
Change 711129 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow-kfserving: add quoting and refactor storage_init limits
Change 711151 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/production-images@master] kubeflow: add wmf-certificates to the storage-initializer
Change 711151 merged by Elukey:
[operations/docker-images/production-images@master] kubeflow: add wmf-certificates to the storage-initializer
Change 711129 merged by Elukey:
[operations/deployment-charts@master] kubeflow-kfserving: update chart
Change 711579 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/production-images@master] kubeflow: add workaround for TLS validation in storage-initializer
Change 711579 merged by Elukey:
[operations/docker-images/production-images@master] kubeflow: add workaround for TLS validation in storage-initializer
Change 712118 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/production-images@master] kubeflow: add the AWS_DEFAULT_REGION env variable to storage-initializer
Change 712118 merged by Elukey:
[operations/docker-images/production-images@master] kubeflow: add the AWS_DEFAULT_REGION env variable to storage-initializer
Change 712346 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow: update storage-init's image and add variable for local gw
Change 712346 merged by Elukey:
[operations/deployment-charts@master] kubeflow: update storage-init's image and add variable for local gw
We have finally something working!
elukey@ml-serve-ctrl1001:~$ curl http://ml-serve1001.eqiad.wmnet:8081/v1/models/enwiki-goodfaith:predict -X POST -d @input.json -i -H "Host: $SERVICE_HOSTNAME"
HTTP/1.1 200 OK
content-length: 112
content-type: application/json; charset=UTF-8
date: Thu, 12 Aug 2021 13:27:13 GMT
server: istio-envoy
x-envoy-upstream-service-time: 31933{"predictions": {"prediction": true, "probability": {"false": 0.03387957196040836, "true": 0.9661204280395916}}}Change 714534 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow-kfserving: add missing comma
Change 714534 merged by Elukey:
[operations/deployment-charts@master] kubeflow-kfserving: add missing comma
Change 714773 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/production-images@master] kubeflow: change storage-init's AWS_DEFAULT_REGION value
Change 714773 merged by Elukey:
[operations/docker-images/production-images@master] kubeflow: change storage-init's AWS_DEFAULT_REGION value
Change 714800 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow: add override in admin_ng for storage-init
Change 714800 merged by Elukey:
[operations/deployment-charts@master] kubeflow: add override in admin_ng for storage-init
Change 715042 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow: raise cpu limits for the kfserving controller
Change 715042 merged by Elukey:
[operations/deployment-charts@master] kubeflow: raise cpu limits for the kfserving controller
Change 715747 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kubeflow-kfserving-inference: add Secret specs for Swift
Change 715747 abandoned by Elukey:
[operations/deployment-charts@master] kubeflow-kfserving-inference: add Secret specs for Swift
Reason:
This shouldn't be needed if I got https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/719128 right