Page MenuHomePhabricator

🟢 Investigating ArgoCD
Closed, ResolvedPublic

Description

note: re-titled because the work on this ticket needed to turn into more of an investigation; findings are documented here, as well as possible next steps.


Deploy https://argo-cd.readthedocs.io/en/stable/ to all the local environment and find pros/cons for local vs external cluster approach.

Have it present and perhaps deploying a "helloworld" application.

n.b. the bootstrapping of this process must be well documented.
We should also write and ADR about this.

Event Timeline

Tarrow renamed this task from Deploy ArgoCD to 🟢 Deploy ArgoCD.Nov 16 2023, 2:29 PM

On Mon 27th Nov 2023 I discussed first steps with @Tarrow

Ideas about possible deployments:

  • A single ArgoCD instance in it’s own cluster
  • One ArgoCD instance on every deployment stage: local, staging, production

Questions:

  • Can and should ArgoCD deploy itself?

Conclusions:

  • We want to keep using helm charts and PR merges in git should trigger deployments
  • It’s hard to come up with a game plan on re-inventing our deployment workflow with ArgoCD before learning about ArgoCD itself and get some experience with it
  • Therefore, as a first step we want to deploy and try it out on the local stage only at first
    • Try to find (dis)advantages of using the same vs external cluster
  • Best case: we can slowly move one component deployment at a time from helmfile to ArgoCD
    • Ideal: current helmfile deployment remains usable during and after the process

Here are some notes and findings after some time of reading through the ArgoCD documentation and trying it out a little bit. I definitely can recommend the docs, especially this overview of core concepts and the operator manual.

Quickstart

The getting started guide can pretty much be followed, but I will quickly note what I did:

  • I used my existing local minikube cluster; but a new empty one works just as fine
  • Created the argocd namespace and installed it via:
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

note: the guide also features manifest for ArgoCD Core, which is a headless deployment missing the UI and API. This actually might be interesting for us at a later stage, but I'd not worry too much about it currently.

  • Installed the CLI tool via brew brew install argocd
  • Port forwarded the UI via kubectl port-forward svc/argocd-server -n argocd 8080:443
  • Generated an admin password via argocd admin initial-password -n argocd

Then you are ready to use it via https://localhost:8080. There is a repository with example apps which I forked and played around with, but I also realized you can simply deploy the wikibase.cloud UI for example since we already have helm charts for it. I did this via the ArgoCD UI. To access that deployment in the browser I also used port forwarding (something like kubectl port-forward svc/wbaas-ui 8001:80 to access it via http://localhost:8001).

single instance vs multiple instances

While it’s probably a good idea to deploy ArgoCD to all stages (local, staging, production) as a first step, it is noteworthy that it is possible to use ArgoCD to deploy to an external k8s cluster, meaning: a cluster that is not the one where ArgoCD itself is deployed.

Possible advantages:

  • A single instance has knowledge about the staging and production cluster at the same time: this opens up some convenient interaction possibilities, like for example blocking deployments of releases to production if they failed to deploy to the staging environment
  • Moving to/Choosing a hosted offering could be simpler
Can and should ArgoCD deploy itself?

It is possible to run a self-managed instance of ArgoCD: https://medium.com/devopsturkiye/self-managed-argo-cd-app-of-everything-a226eb100cf0

I'm intrigued by the idea, but I advise against it as part of our first steps for transitioning our deployments to ArgoCD. I think at a later point in time it is easier to revisit this idea, once the wikibase.cloud engineers gained some more experience and understanding of how they want to use ArgoCD as part of wikibase.cloud's architecture. In the meantime it would be beneficial to deploy ArgoCD with the (in comparison) well understood helmfile approach, for the sake of stability and transitioning deployments. In that sense the ArgoCD deployment can be thought of as the last deployment that should be moved over to the new ArgoCD deployment workflow, and if it should be moved at all can be decided at a later point in time.

ArgoCD Image Updater

https://argocd-image-updater.readthedocs.io/en/stable/

A tool to automatically update the container images of Kubernetes workloads that are managed by Argo CD.

I tried setting this up but couldn’t really get it to work. While reading the documentation I noticed several reasons why we might don’t want/need this though:

  1. While being actively worked on, the docs state that the tool isn’t considered production ready: “You are welcome to test it out on non-critical environments, and of course to contribute by many means. There will be (probably a lot of) breaking changes from release to release as development progresses until version 1.0”
  2. Rollbacks aren’t currently supported https://argocd-image-updater.readthedocs.io/en/stable/basics/update/#rollback-and-image-updates
  3. It might not be the right tool for the wikibase.cloud use case, since deployments via merging PRs seem more desirable for safety and coordination reasons
Further notes

While reading the docs I noticed some things we probably want to keep in mind going forward:

  1. TLS: ArgoCD utilizes inbound TLS - out of the box uses a self-signed certificate but for actual deployments we need to configure this properly: https://argo-cd.readthedocs.io/en/stable/operator-manual/tls/
  2. HA: Since we want to rely on ArgoCD in the future, we should look into setting it up the High Availability version of ArgoCD
  3. To achieve a gitops workflow, instead of manually configuring ArgoCD & our deployments (or "Applications" in ArgoCD speak) we want to do it declarative via git: https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/
  4. I noticed that custom styling is supported - at some point this would be very nice to utilize in order to be able to quickly distinguish between the staging vs. production instance: https://argo-cd.readthedocs.io/en/stable/operator-manual/custom-styles/
Next steps

Possible next steps/tickets could be (to be discussed in technical refinement):

  1. Draft a desired deployment workflow (requirements, wants & needs)
  2. Integrate the ArgoCD charts into the existing wbaas-deploy helmfile setup for the local environment
Deniz_WMDE renamed this task from 🟢 Deploy ArgoCD to 🟢 Investigating ArgoCD.Dec 6 2023, 5:15 PM
Deniz_WMDE removed Deniz_WMDE as the assignee of this task.
Deniz_WMDE updated the task description. (Show Details)
Deniz_WMDE moved this task from Done to In Review on the Wikibase Cloud (Kanban board Q4 2023) board.
Deniz_WMDE updated the task description. (Show Details)
Deniz_WMDE subscribed.
Tarrow claimed this task.