Page MenuHomePhabricator

Helm packages deployment tool, at least for cluster applications.
Closed, ResolvedPublic

Description

We need a tool capable of deploying an application set with a specific configuration in a given cluster. This tool needs to fulfill the following requirements:

  • should deploy applications packaged as helm charts.
  • should handle secrets on those applications.
  • should manage environments (config per DC, config per cluster)
  • every config change should be kept on git.

A small list of tooling that could meet this set of requirements:
https://github.com/Praqma/helmsman
https://github.com/roboll/helmfile
https://github.com/Eneco/landscaper

Details

Related Gerrit Patches:

Event Timeline

fsero triaged this task as Medium priority.Dec 17 2018, 2:56 PM
fsero created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 17 2018, 2:56 PM
fsero added a comment.Feb 26 2019, 4:32 PM

we are going to pick helmfile for now as it seems to have an slighly wider community. I'll work on create a suitable package for us for helmfile

Change 517887 had a related patch set uploaded (by Fsero; owner: Fsero):
[operations/deployment-charts@master] introducing helmfile.d values for staging cluster

https://gerrit.wikimedia.org/r/517887

Change 517888 had a related patch set uploaded (by Fsero; owner: Fsero):
[operations/puppet@production] k8s, deploy: introducing helmfile for manage charts

https://gerrit.wikimedia.org/r/517888

fsero claimed this task.Jun 21 2019, 10:47 AM
fsero moved this task from Backlog to Doing on the serviceops board.

Change 517887 merged by Fsero:
[operations/deployment-charts@master] introducing helmfile.d values for staging cluster

https://gerrit.wikimedia.org/r/517887

Change 517888 merged by Fsero:
[operations/puppet@production] k8s, deploy: introducing helmfile for manage charts

https://gerrit.wikimedia.org/r/517888

Change 520387 had a related patch set uploaded (by Fsero; owner: Fsero):
[operations/puppet@production] helmfile,k8s: cannot apply helm secrets due to missing user

https://gerrit.wikimedia.org/r/520387

Change 520390 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] helmfile: Set owner for secrets values

https://gerrit.wikimedia.org/r/520390

Change 520387 abandoned by Fsero:
helmfile,k8s: cannot apply helm secrets due to missing user

Reason:
better approach https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /520390

https://gerrit.wikimedia.org/r/520387

Change 520390 merged by Fsero:
[operations/puppet@production] helmfile: Set owner for secrets values

https://gerrit.wikimedia.org/r/520390

fsero added a comment.Jul 3 2019, 4:52 PM

pending some documentation for helping people to migrate this is essentially done

fsero added a comment.Jul 5 2019, 1:04 PM

after further testing it seems that in order to use helmfile we need to set up some environment variables i.e HELM_HOME=/etc/helm KUBECONFIG=/etc/kubernetes/zotero-staging.config helmfile diff

HELM_HOME variable is needed because we have overriden[[ https://github.com/wikimedia/puppet/blob/d18a64085ca31d2e87f8f66328aa479b10c5d61a/modules/helm/manifests/init.pp#L37 | stable repo to point ]] to the wmf releases repo.
HELM_HOME variable could be avoided using --home flag on helm and helm-diff, and while helmfile offers and "args" field to include extra args it add them at the end of the command line where --home flag is not honored.

we need KUBECONFIG variable to use the right token for the combination of service and cluster. While helmfile and helm also supports a --kubeconfig flag where we can pass the path and kept it in code. However helm-diff plugin does not support --kubeconfig flag and creating a diff fails.

So in order to avoid both variables helmfile and helm-diff should be patched upstream. While the patch is not merged or the issue fixed upstream we can circumvent it creating a source file (lets say i named .hf_env) and call helmfile this way:

source .hf_env; helmfile diff

Change 521441 had a related patch set uploaded (by Fsero; owner: Fsero):
[operations/puppet@production] helmfile,k8s: creating hfenv variables

https://gerrit.wikimedia.org/r/521441

Change 521441 merged by Fsero:
[operations/puppet@production] helmfile,k8s: creating hfenv variables

https://gerrit.wikimedia.org/r/521441

Change 521505 had a related patch set uploaded (by Fsero; owner: Fsero):
[operations/puppet@production] helmfile,k8s: bug: we should require the directory if not fails

https://gerrit.wikimedia.org/r/521505

Change 521505 merged by Fsero:
[operations/puppet@production] helmfile,k8s: bug: we should require the directory if not fails

https://gerrit.wikimedia.org/r/521505

Change 521512 had a related patch set uploaded (by Fsero; owner: Fsero):
[operations/puppet@production] helmfile,k8s: bug: we should require the directory if not fails

https://gerrit.wikimedia.org/r/521512

Change 521512 merged by Fsero:
[operations/puppet@production] helmfile,k8s: bug: we should require the directory if not fails

https://gerrit.wikimedia.org/r/521512

Change 521836 had a related patch set uploaded (by Fsero; owner: Fsero):
[operations/puppet@production] k8s: putting a deprecation notice on scap-helm

https://gerrit.wikimedia.org/r/521836

Change 521836 merged by Fsero:
[operations/puppet@production] k8s: putting a deprecation notice on scap-helm

https://gerrit.wikimedia.org/r/521836

fsero closed this task as Resolved.Jul 10 2019, 10:35 AM

Change 522098 had a related patch set uploaded (by Fsero; owner: Fsero):
[operations/deployment-charts@master] helmfile.d: adding eqiad,codfw admin helmfiles

https://gerrit.wikimedia.org/r/522098

Change 522098 merged by Fsero:
[operations/deployment-charts@master] helmfile.d: adding eqiad,codfw admin helmfiles

https://gerrit.wikimedia.org/r/522098

Change 540843 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Fully remove scap-helm

https://gerrit.wikimedia.org/r/540843

Change 540843 merged by Alexandros Kosiaris:
[operations/puppet@production] Fully remove scap-helm

https://gerrit.wikimedia.org/r/540843