Page MenuHomePhabricator

Fix Archived Bitnami Chart References
Closed, ResolvedPublic

Description

This was seen when running helmfile deps:

in ./helmfile.yaml: unable to update deps: no resolved dependency found for "redis", running "helmfile deps" may resolve the issue

It seems that the resolution of T401478 didn't go quite far enough. We did create copies of the git repository but unfortunately the index.yaml that we copied here still referenced artifacts from bitnami servers which are now missing. This ticket is an attempt to fix that by now referencing chart packages we have built from source and archived ourselves. We noticed this issue in the context of T385981 where it appears a redis chart was missing.

@Ollie.Shotton_WMDE
we previously used a Helm repository that was an index.yaml file in a Git repo that we forked
that index.yaml file pointed at packages and containers hosted by Bitnami not ones we controlled
that caused us to not be able to pull a redis chart when trying to test the MW1.43 update
to (attempt to) fix this, we created the chart packages ourselves and hosted them as GitHub release assets on our fork of the Bitnami git repo and used direct links in helmfile.yaml rather than a Helm repository

PRs:

Event Timeline

The mariadb, elasticsearch and nginx charts now got fixed - what's left is to do the same for redis and redis-2 deployments. It seems that it's not possible to provide a targz chart archive to ArgoCD, so I tried to add the 2 redis charts to our own chart repository. This also didn't work out in the end, but I think it's only due to some minor oversight in the test configuration I tried (https://github.com/deer-wmde/debug-charts)

The error message I get is Failed sync attempt to : one or more objects failed to apply, reason: Application.argoproj.io "redis" is invalid: spec.sources[0].repoURL: Required value,Application.argoproj.io "redis-2" is invalid: spec.sources[0].repoURL: Required value , although in both redis configurations the repoURL is specified (even hardcoded, for testing purposes) so there seems to be a missing puzzle piece: https://github.com/deer-wmde/debug-charts/blob/main/charts/argocd-apps/templates/redis.yaml#L14

Today I got it mostly working, although there was some confusing error with the redis-2 deployment. In the daily today I said if I can't easily solve this today I think we should just move the redis deployments back to the helmfile again for now, so it does not block us anymore. After this little odyssey I think overall the engineers should discuss if it would make sense to entirely move back to helmfile, as the benefits from argocd so far are marginal in comparison to the complexity overhead.

@Andrew-WMDE and I coordinated the deployment of these change together today

  • A few more PRs were needed because ArgoCD wasn't happy about the allowed bitnami repoURL getting removed before the releases were
  • Getting rid of the ArgoCD redis deployments required to trigger a manual sync with the prune flag, as per the current configuration
  • Updating the releases in ArgoCD and in the helmfile was done in 2 steps by making use of the a installed: false attribute in helmfile
    • This also made us realize that there apparently was some remnant of the once-by-helmfile deployed redis deployment, which stirred some confusion
  • => Ultimately we're on a clean cluster state again though for staging and production, and both redis deployments are now handled by helmfile for now

After this little odyssey I think overall the engineers should discuss if it would make sense to entirely move back to helmfile, as the benefits from argocd so far are marginal in comparison to the complexity overhead.

I think this idea does need some investigation; definitely as to if argo is the right choice or not for us. I think we also need to consider if staying with helmfile is sustainable too.

However, I think something that jumped to my mind when I came back to this ticket is that probably just hosting a raw .tgz file is a pretty non-standard way to ship about helm charts and therefore it's not super surprising that the industry standard tooling doesn't support easily doing stuff in a non-standard way.

Trying to do this is all a band-aid for trying to keep this old archived chart in use rather than updating and moving forwards (probably now to even a new KV store; valkey?)