On Mon 27th Nov 2023 I discussed first steps with @Tarrow

Ideas about possible deployments:

• A single ArgoCD instance in it’s own cluster
• One ArgoCD instance on every deployment stage: local, staging, production

Questions:

• Can and should ArgoCD deploy itself?

Conclusions:

• We want to keep using helm charts and PR merges in git should trigger deployments
• It’s hard to come up with a game plan on re-inventing our deployment workflow with ArgoCD before learning about ArgoCD itself and get some experience with it
• Therefore, as a first step we want to deploy and try it out on the local stage only at first
  • Try to find (dis)advantages of using the same vs external cluster
• Best case: we can slowly move one component deployment at a time from helmfile to ArgoCD
  • Ideal: current helmfile deployment remains usable during and after the process

Here are some notes and findings after some time of reading through the ArgoCD documentation and trying it out a little bit. I definitely can recommend the docs, especially this overview of core concepts and the operator manual.

Quickstart

The getting started guide can pretty much be followed, but I will quickly note what I did:

• I used my existing local minikube cluster; but a new empty one works just as fine
• Created the argocd namespace and installed it via:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

note: the guide also features manifest for ArgoCD Core, which is a headless deployment missing the UI and API. This actually might be interesting for us at a later stage, but I'd not worry too much about it currently.

• Installed the CLI tool via brew brew install argocd
• Port forwarded the UI via kubectl port-forward svc/argocd-server -n argocd 8080:443
• Generated an admin password via argocd admin initial-password -n argocd

Then you are ready to use it via https://localhost:8080. There is a repository with example apps which I forked and played around with, but I also realized you can simply deploy the wikibase.cloud UI for example since we already have helm charts for it. I did this via the ArgoCD UI. To access that deployment in the browser I also used port forwarding (something like kubectl port-forward svc/wbaas-ui 8001:80 to access it via http://localhost:8001).

single instance vs multiple instances

While it’s probably a good idea to deploy ArgoCD to all stages (local, staging, production) as a first step, it is noteworthy that it is possible to use ArgoCD to deploy to an external k8s cluster, meaning: a cluster that is not the one where ArgoCD itself is deployed.

Possible advantages:

• A single instance has knowledge about the staging and production cluster at the same time: this opens up some convenient interaction possibilities, like for example blocking deployments of releases to production if they failed to deploy to the staging environment
• Moving to/Choosing a hosted offering could be simpler
  • https://akuity.io/
  • https://www.koncrete.dev/

Can and should ArgoCD deploy itself?

It is possible to run a self-managed instance of ArgoCD: https://medium.com/devopsturkiye/self-managed-argo-cd-app-of-everything-a226eb100cf0

I'm intrigued by the idea, but I advise against it as part of our first steps for transitioning our deployments to ArgoCD. I think at a later point in time it is easier to revisit this idea, once the wikibase.cloud engineers gained some more experience and understanding of how they want to use ArgoCD as part of wikibase.cloud's architecture. In the meantime it would be beneficial to deploy ArgoCD with the (in comparison) well understood helmfile approach, for the sake of stability and transitioning deployments. In that sense the ArgoCD deployment can be thought of as the last deployment that should be moved over to the new ArgoCD deployment workflow, and if it should be moved at all can be decided at a later point in time.

ArgoCD Image Updater

https://argocd-image-updater.readthedocs.io/en/stable/

A tool to automatically update the container images of Kubernetes workloads that are managed by Argo CD.

I tried setting this up but couldn’t really get it to work. While reading the documentation I noticed several reasons why we might don’t want/need this though:

1. While being actively worked on, the docs state that the tool isn’t considered production ready: “You are welcome to test it out on non-critical environments, and of course to contribute by many means. There will be (probably a lot of) breaking changes from release to release as development progresses until version 1.0”
2. Rollbacks aren’t currently supported https://argocd-image-updater.readthedocs.io/en/stable/basics/update/#rollback-and-image-updates
3. It might not be the right tool for the wikibase.cloud use case, since deployments via merging PRs seem more desirable for safety and coordination reasons

Further notes

While reading the docs I noticed some things we probably want to keep in mind going forward:

1. TLS: ArgoCD utilizes inbound TLS - out of the box uses a self-signed certificate but for actual deployments we need to configure this properly: https://argo-cd.readthedocs.io/en/stable/operator-manual/tls/
2. HA: Since we want to rely on ArgoCD in the future, we should look into setting it up the High Availability version of ArgoCD
   • https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/
3. To achieve a gitops workflow, instead of manually configuring ArgoCD & our deployments (or "Applications" in ArgoCD speak) we want to do it declarative via git: https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/
4. I noticed that custom styling is supported - at some point this would be very nice to utilize in order to be able to quickly distinguish between the staging vs. production instance: https://argo-cd.readthedocs.io/en/stable/operator-manual/custom-styles/

Next steps

Possible next steps/tickets could be (to be discussed in technical refinement):

1. Draft a desired deployment workflow (requirements, wants & needs)
   • possibly useful article & discussion: https://buildkite.com/blog/applying-sre-principles-to-cicd
2. Integrate the ArgoCD charts into the existing wbaas-deploy helmfile setup for the local environment
   • https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd