Spike summary

I explored with adjusting the k8 workshop to Apache Flink. It boils down to running Flink on minikube. This can be done locally, without the need of a cloud vps vm.

Following are some consideration to bring into the next grooming seession.

I'd say that Could VPS would not buy us much, other than _potentially_ granting multi users access to a self-hosted minikube - or expose a public facing service. I don't think we want to go down the path of maintaining either (for dev workflows).

Setting up minikube is a well documented and straightforward process (at least on macOS/linux).
For running Flink on k8, I explored two paths:

1. Adjusting the Search flink-session-cluster helm charts.
2. Using the recently release Apache Flink Kubernetes Operator.

While for production use cases we should clearly adopt 1), both approaches offer interesting angles for experimentation and local development.

Path 1) requires a Docker image and decoupling the charts from the specific use case and WMF envs (https://github.com/wikimedia/operations-deployment-charts/blob/master/charts/flink-session-cluster/values.yaml). We should consider contributing to a generic enough config, and make the setup more self service for developers (that want to run things on minikube).

Path 2) was easier to setup "out of the box". Setting up Cluster deployments that can accept Job submission either interactively or programmatically is well documented https://github.com/apache/flink-kubernetes-operator/tree/main/examples. The tutorial at
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/ gives the basic building blocks for setting up a Flink Cluster ready to accept jobs.

For reference, some resources on how Google and Spotify are operating Flink on k8:

• https://github.com/GoogleCloudPlatform/flink-on-k8s-operator
• https://github.com/spotify/flink-on-k8s-operator

@gmodena thanks for exploring these k8s deployment options!
Something I used to test H/A capabilities (restarts&recovery) was https://min.io/ with minikube, I might still have some config examples and I remember it was not quite trivial to setup, but most probably because of my lack of knowledge of k8s.
Making the current flink-session-cluster helm chart more generic is definitely something that sounds valuable in the short/mid-term.
For the long term I wish we can explore using the apache flink-kubernetes-operator in production, the hope is that it could solve some the pain points we have regarding k8s and job management.