Page MenuHomePhabricator

Better scaffolding for helm charts / releases
Open, MediumPublic

Description

PROBLEM STATEMENT

Right now, the process of scaffolding charts "works", but produces charts like https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/726933 which are overcomplex and often include stuff that is not really needed for the chart to work - like all the list of containers for both nodejs and php deployments.

Moreover, scaffolding still doesn't include the option to generate a cronjob, which are increasingly used in production.

Finally, scaffolding doesn't provide any form of help in creating an helmfile.d deployment.

As an additional dimension to the problem: scaffolding is now written in ruby, as part of our Rakefile, to keep the dependencies of running it to a minimum. It creates an issue of maintainability.

PROPOSED SOLUTION
We should write a wizard, whose logic can be plugged both in a cli tool and served via web, that allows to generate a tailor-made chart with only the components needed by the user at the present moment.

This wizard should do the following:

  • Ask for a name for the chart ("what software do we want to run?")
  • Ask for the service port to be reserved on https://wikitech.wikimedia.org/wiki/Kubernetes/Service_ports if this is a production chart
  • Ask what language is the software written in (php/nodejs/python/golang for now?)
  • Asks if prometheus support is native or not
  • Asks if some additional sidecars are needed (mcrouter/nutcracker/logging)
  • Asks if running periodic jobs or one-off jobs is needed
  • Finally, asks for the image name and tag

and generates a chart based on those answers, only including the needed pieces.

For added maintainability, we should add comments in the templates referencing the version/sha1 of the scaffolding, so that we can easily find specific templates to update with a git grep.

Event Timeline

Joe triaged this task as Medium priority.Oct 8 2021, 8:47 AM

To be honest, I am not understanding what the solution exactly is. I gather that the point is to not have a lot of boilerplate code in the charts? Because the rest sounds a lot like what we currently have. We already have a wizard. It may be not asking all these questions (and I am of the opinion it should NOT be asking some of these questions e.g. mcrouter or not), but it's asking already 3 of them. We can add a few more with sane defaults.

Now, the generated charts are large, but that's mostly because of duplication and could be greatly simplified by moving more things to common_templates/ and if guarding them. That would make reviews way easier (knowing that a chart just uses parts of already vetted common_templates would definitely be a plus)

Note that generating fully custom tailored charts poses the inverse problem. When the time comes to add some functionality to the chart, it will have to be done manually by the owner of the chart and will lead to non upgradeable situations as well.

To be honest, I am not understanding what the solution exactly is. I gather that the point is to not have a lot of boilerplate code in the charts? Because the rest sounds a lot like what we currently have. We already have a wizard. It may be not asking all these questions (and I am of the opinion it should NOT be asking some of these questions e.g. mcrouter or not), but it's asking already 3 of them. We can add a few more with sane defaults.

Now, the generated charts are large, but that's mostly because of duplication and could be greatly simplified by moving more things to common_templates/ and if guarding them. That would make reviews way easier (knowing that a chart just uses parts of already vetted common_templates would definitely be a plus)

Note that generating fully custom tailored charts poses the inverse problem. When the time comes to add some functionality to the chart, it will have to be done manually by the owner of the chart and will lead to non upgradeable situations as well.

These are all good points and I mostly agree, let me explain more what I'd like to do:

  • Have a lot more functionality baked into common templates, broken down in smaller files. So for instance, if you want to add mcrouter to your chart, to follow the example above, it should definitely come as linked from common_templates
  • Probably move the versioning from the top level of common templates to the specific module, so instead of common_templates/0.4/_tls_helpers.tpl we could have something like common_templates/tls/helpers_0.4.tpl
  • The idea of using a wizard to compose the chart was mainly to allow people to pick pieces of the standard chart, even those that might need more diversification and so will need to be copied over from scaffolding, to be tailored to the needs of the user and to be marked clearly for upgrade.

We can just get away with using our current wizard, and just work on the common_templates structure and improve the scaffolding chart. To be honest, I like when people get from scaffolding just what they strictly need.

I'll try to write down some of the ideas I have in a patch.