Page MenuHomePhabricator

Evaluate cost/benefits of switching to using a programming language to define kubernetes resources
Open, Needs TriagePublic


Helm is a nice tool, and with helmfile it becomes manageable to deploy in production via git. It has some big outstanding issues:

  • You write templates that output yaml. That's easy to get wrong, and ugly ({{ include tpl . | indent 6 }} anyone?)
  • Reusability is very limited. We had to come up with tricks and symlinks to use the same yaml bits in multiple charts
  • It's hard to control centrally. Even with helmfile, propagating changes across charts is mostly a manual process, which is a pain fist and foremost for SREs
  • The charts are hard to grasp and (I suppose and hope) mostly obscure to our developers.
  • Go text/template is not the nicest templating system on the planet

I've taken a look at two solutions for managing kubernetes via code, cdk8s and pulumi.

The two projects are very similar:

  • both allow to write code to generate a kubernetes resource instead of yaml
  • both are polyglot, allowing us to use python or typescript. Pulumi also supports golang, which I consider a bad choice in this case though. My exploration was only based on the python interface.

there are some differences though:

  • cdk8s' python is very structured, with every field having a specific type, and it has already some abstraction on top of the k8s data structures. Pulumi takes instead the road of letting you define a full nested dictionary with all the key/values that will end up in the yaml, more or less
  • pulumi includes a deployment tool, while cdk8s just generates k8s yaml
  • pulumi can also deploy helm charts
  • pulumi has a configuration engine that stores values $somewhere, while in cdk8s all that layer would need to be implemented.

Overall, I found cdk8s to have the better python abstraction, at least at first sight, so I concentrated my effort on it, even if it's less feature-complete and would need us to write some tooling or adapt cdk8s to be used with some kubernetes deployment tool like helm.

From a first stub at implementing a chart for blubberoid with cdk8s, I can say it would enable us, if well used, to solve most of the above grievances we currently have with helm.

Turning it into a comparable product would mean though that we'd need to dedicate some engineering time to it.

I opened this task to discuss if some solution like the ones mentioned above seems interesting enough to us to take the effort to move our deployment to such a system.

Event Timeline

Joe created this task.Jun 8 2020, 11:16 AM
Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptJun 8 2020, 11:16 AM
CDanis added a subscriber: CDanis.Jun 8 2020, 3:06 PM
jijiki moved this task from Incoming 🐫 to Unsorted on the serviceops board.Aug 17 2020, 11:45 PM