Page MenuHomePhabricator

Ensure skaffold works smoothly with argo deployed services
Open, Needs TriagePublic

Description

It seems that on clusters that have never had a helm deployment at all skaffold is unable to run in it's helm configuration. This is now the case on fresh development clusters that have only had services deployed using argo-cd

Workarounds
A quick work around is available at https://github.com/wmde/wbaas-deploy/pull/1955 which uses raw kubectl. Unfortunately this work around only works with changes to the image and doesn't include updates to the charts.

Another work around recommended by @dena is to make helm deployment happen is to:

Re-Add the API helm release back into helmfile.yaml: https://github.com/wmde/wbaas-deploy/blob/main/k8s/helmfile/only-for-argo-value-generation.yaml#L51-L55
Since this release will exist before the one that Argo deploys, it will be possible to replace via skaffold
Re-Create your local cluster
Run make skaffold-api

Event Timeline

Tarrow added a subscriber: dena.

@dena what do you mean exactly by "Re-Create your local cluster"? I tried make apply-local but that didn't work due to helm complaining that a resource already exists. I managed to manually delete all the resources that were already existing and then make apply-local worked. Are you suggesting doing a make minikube-delete and recreate the cluster from scratch again? Thanks

Are you suggesting doing a make minikube-delete and recreate the cluster from scratch again?

Yes exactly. It's tedious but I happen to do it rather often, hence at some point I created this script here for convenience which recreates it in an unattended fashion: https://github.com/wmde/wbaas-deploy/blob/main/bin/local/new-local-cluster.sh
On my machine it usually takes like ~16 minutes. Sometimes the creation of resources runs into race conditions and doesn't resolve properly (Error: namespaces "api-jobs" not found for example) but just running it again a second time usually works then for me.

This number of years on, I do still wonder how the team feels about using skaffold as the primarily local dev environment
It was primarily chosen / created during T292830: [INITIATIVE 1.5] Setup local cluster at the back of 2021 and i believe the decisions are likely in some google docs not on phab.
I only ask because personally the words tedious and ~16 minutes around the topic of development environments always make me shudder slightly, and I was there in during the decision and wonder how team needs have evolved or solidified since then

Thanks for that information @dena!

@Addshore Personally, I'm not fixed on Skaffold. However, is this issue due to the way we currently have Skaffold and/or ArgoCD setup, or because Skaffold is the wrong tool for the job? Skaffold seems to work okay for helmfile (rather than ArgoCD) deployed releases, and I wouldn't want to throw the baby out with the bathwater. Do you have any suggestions as to what we could use instead of Skaffold?